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Foreword 

Somewhere, one of the sages has written, "Nothing is so powerful 
as an idea w hose time has come." This statement applies clearly to the IRA 
Board's \otingin 1965 to establish ihc Reading Research Qiiarlerly. 

True, we had an alert and perceptive Board — one which could take 
a long-term view and see that the Association had the clear oblig«ition to 
provide an official ira outlet for serious, length) papers which explored 
research problems deep!) and insightfully — problems which were practi- 
cal, theoretical, and sophisticated. 

As president of the Associ.ition at the time of the approval of the 
foundingof ihcQjuarierly, I will admit to "good words in its fa\or spoken in 
smoke-filled i ooins." Although at times I ha\ e heard Ralph Staiger's mind 
click like a taxi meter — and aren't we fortunate for his long guidance in 
financial and professional matters?— he did support the idea of ihc Quar- 
terly, which was likely to appear on the negative side of the Association's 
financial ledger. 

So a happy circunistance of the times, an enlightened Board and a 
sympathetic Execuii\e Director, led to the founding of ira's Uiird journal 
and also set the Association's cum se on a publication program which is 
today mjissive, diversified, and distinguished. 

As the first editor of the Quarterly, I am proud of how far we have 
come, of the excellent subsequent editors who impro\ed and expanded 
oui original concept of the Q'^^r/^ r/), And I am pt oud that the Association 
has accepted its responsibility to ser\e the members with a variety of 
publications. 

Ten years from now we will have mo\ed in new % important direc- 
tions in all of our publications, which w ill ha\e e\en greater usefulness to 
OUI members. There is no cjuestion of this. There can be no question of 
this. 

The series of papers of this \olume from the New Orleans Precon- 
vention Insuiute on Research result in a document which merits the 
attention of all members of the Association. It marks the tenth \olume 
yeai of the Quarterly and is anothei significant contribution to the litera- 
ture by IRA. 

'I'HEODORE CLVMER 

Founding Editor, Reading Research Quarterly 
Director, Institute for Reading Research, 

Santa Barbara 
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SAMUEL WEINTRAUIi 

Siaie Vniversity of New York at Buffalo 

ROGER FARR 

Indiana University 



This \oliiine on improving le.iding research focuses on basic de- 
sign questions which too often flaw research. A lack of attention to 
fundamentals — not to esoteric or ,sophistic<ited concerns — accounts for 
much that h*is been we.ik in research, These basics should be second 
nature to the experienced researcher, but too often they are not. And they 
should be emphasized for the neophyte researcher so that he can sharpen 
liisskills.The purposeof this volume is to focus on the fundamentals in an 
effort to improve future research conducted by both the experienced 
researcher and the neophyte. 

A review of numerous studies on who conducts research indicates 
that fre(|uently a researchei conducts only one study . Too often the (juest 
of the budding investigator gets nipped, and the dissertation stands «is the 
sole research contribution. Occasionally a research effort follows the 
dissertation, but then the investigatoi^s name disappears from the re- 
search literature. The bulk of reseaich, then, is done by one-time re- 
seaichers, and the members of the re.iding profession who continuously 
pursue and report research efforts constitute a distressingly small 
number. 

One would .issume that the experienced researcher would produce 
a better effort than die neophyte. At least it makes sense that experience 
in any area is a powerful teachei . \o one has really studied this thesis, and 
so we do not hwiv that such is the case. It m«iy be that if the beginning 
reseaichei is careless he continues to produce sloppy efforts, showing 
little oi no impioveinent, but one would hope that one way to improve 
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research w ould be (In oiigh continued practice in doing it u hile sharpen- 
ing the researchers thinking skills. 

Another avenue toward impro\ing research efforts is to make the 
first efforts of the neophyte better. Even if it is his only effort, a general 
improvement of reading research would result. This \olume should be 
valuable foi iUc beginning researcher. That includes the graduate stu- 
dent who is in search of a dissertation topic as well as the student at die 
stage of de\eloping his proposal and the student preparing to write his 
final report. It includes the post doctoral researchei thinking about his 
first independendy planned research and perhaps e\en the more sophis- 
ticated worker who has alieady produced several in\estigations but still 
feels the need to sharpen his basic research skills. The volume is addressed 
to all researchers who understand the need to make their efforts more 
credible. 

Does reading research need to improve? 

Since William S. Gray first began abstracting and annotating the 
reading research literature in 1925, about 9,000 published reports ha\e 
been identified. Itwuuld seem thate\ery aspectof reading has surely l>een 
investigated with marked thoroughness. And certainly with so many 
published reports extant, all of oui (|uestions should ha\elxien answered. 

Regrettably, such is not the case; unanswered questions abound. 
And criticisms of the research are not at all unusual. Gray (1950) noted 
that the research was fragmentary. In re\iewing the publications on 
phonic progiams, Chall (1967) remarked that the research in that area 
was disappointingly inconclusi\e. She attributed hei conclusion to die 
poor {|uality of the research she re\icwed. Others have made similar 
statements relati\e to reading research, and the documentation of such 
statements would include dozens of references. 

The Readiug Research Qiimterly annually accepts lor publication 
about one out of every 8-10 manuscripts received. Most of those not 
published are rejected because of a lack of attention to the fundamentals 
of good research. The design may be inappropriate, the statistical 
techniques incorrect, the tests in\alid, the sample poor, or the study so 
basically flawed in any numbei of other ways as tolx:of\ery limited value. 
Some of the studies rejected by the Qiiarterly have appeared in print 
elsewhere without ha\ing been impro\ed upon — their flaws remain. 

The continued publication of seriously flawed research probably 
does more harm than good to the profession. Although a poor study may 
have one or more elements of merit, it is, ne\ertheless, still poor and 
probably should not appear in print— at least not w ithout an accompany - 
ingcriti{|ue. The editors of joiiiiials and the advisory board inemlx^rs on 
refercedjomnals share the responsibility for seeing that minimum stan- 
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chiixls()r(|ualit} aiciiicl. riic publication of weak research simpi) encoui- 
ages further pruductii)iiuf the same, ' There is no need to niipru\ e il'une is 
rewarded for inappropriate (luestions, flawed designs, and f'u/./} thinking. 

The man) criticisms of published research, the ver) few definite 
conchisioiis that can be drawn fnmi the numerous studies reported, the 
high numlx^r of rejections b) not just ihc Qjnarterl) but other publications, 
all suggest that the need tt) inipH)\e thetpialit) of thcjeseai ch literature is 
not only a worthwhile undertaking but an imperative need. 

What is research! 

In an editorial statement, we defined research as a wa) of thinking 
{Reading Research Qiiarierly, Vol, X, No. I), It is the sharpening and 
clarification of this thinking to which the papeis in this \olume address 
theinsehes, VVaidiop einphasi/.es that good research begins with asking 
giH)d (|uevStions, Asking good i|uestions is anothei wa) of thinking clearl), 
Pel haps w e lia\e doi:e liule of the htttei in our past i eseai ch efforts — oi at 
least not enough of it. 

In her p.ipei , Robinson raises a point that precedes Wardrop's 
need foi asking a good question. She implies that a good research (jues- 
tion can onl) Ix; developed from an e\teiisi\e knowledge base. Good 
research, then, begins with a scliolaii) grasp of an area. It does not 
develop from an intuition oi a visceial feeling, The intuition ma) have 
been the conceptioji, but before the final deli\er) of the worthwhile 
questions, much incubation must take place. Unlike ,\thcna, a stud) does 
not appear full-blown from the head of the icseaichei , The development 
of a worthwhile i|uestion might thus be analogous to the fetal stage of the 
infant. And 9 months fiom conception to deliver) ma) even be an undei- 
estimate of the time needed foi a well-developed (|uestion to be lx)vn. 
Only the individual who knows what has lx;en done can know what )et 
needs to be done, 

Wardhaugli leniinds us that becoming proficient in linguistics is no 
eiisy task; and Goodman suggests that the parallel is true in reading. 
Although mail) authors discourse as if the) wei eexpcits in the aiea, thcii 
understanding is often quite limited, 6'iW leseaich in leading must have 
as its basis a solid undei standing of at least the area undci investigation, 
and the essential lelevance betvveen linguistics and reading exemplifies 
how bettci research can result from a broadei knowledge base from 
several areas in reading, 

.Most of the papers in this volume deal with but one type of 
research — the classical empirical design. This emphasis seems valid be- 
cause the vast piepondeiance of leseaich literature that has been done 
and that continues to Ix: pioduced follows the classical model. The em- 
phasis is upon statistical procedures, and the outline tends to follow 



VV'EINTRAUB AND FARR 



9 



3 




relativcl) well cMablislicd Mages. Tone gives an ovcnie;\ of these steps in 
liis chapter as well as thoughtful considerations for the writing of each. 

Research, however, is— or ought to be — more broadly defined than 
the classical type model. It seems crucial that wc begin to look at other 
types of rescMrch «uid recogni/e ihem as respectable, \vaich-op, Ward- 
liaugli, and Robinson cithei refei to othei types of research or at least 
inipl) the need foi seaiching out othci types. Robinson mentions histori- 
cal and case study, «unong others. We would like to suggest the need for 
going beyond even these recognized — albeit not always **respectable" — 
forms. It IS time that we searched for new models to follow oi developed 
oui own. As noted in anothei editorial statement, methodological incai- 
cenuion has afflicted us [Reading Research Qiuirterly, X, No. 4). 

The fractionation that has developed in our search for the reading 
process and in the various research efforts tliat have attempted to resolve 
some of OUI nietluKlology piobleiiis may Ix; directly attributable to the use 
of iiiappiopiiate research models, In our efforts to gain academic re- 
spectability in the eyes of allied piofessions, we have adopted models of 
leseaich that may be acceptable foi them but are peiliaps not useful and 
indeed may Ik- iptitc hai infill in our ow ii field. The sei ies of papers in this 
volume do not tieat otiici possibilities at all. It is perhaps something that 
ought to Ix: dealt with in anothei \ohiiiie. We do feel that the coiLsidera- 
tioii of altei native models of research as well as the development of 
entirely new ones is an issue that demands the attention of the most 
competent minds in the Held. 

What (mm oj reading research demand emphasisS? 

Linguistics and reading research 

Reading is a broad field related to many otiiei fields. The annual 
suiiiinary of investigations, which appeals as a full issue oi' Reading Re- 
M-anh ()ua)(eri\ each yeai, legularly iiioiiitous jouiiials in such diverse 
arcMs as iieuiology, sociology, learning disabilities, optometry, psychol- 
ogy Jiiiguistius. child development, and fine arts, among others. It v\ould 
ix: appropi iate pei haps to iiiLlude pajxris on the problems of conducting 
invesiigatioiis in e<icli of these areas as they relate to reading. Space 
limitations among other factors militated against this. The current pro- 
ductivity of and inteiest in research on liiiguistiusand leading demanded 
that it be singled out and represented in this volume. 

The importance of linguistics in gaining insights into the reading 
pi ocess cannot be disputed. The need to improve tliet|ualiiy of research 
efforts in this aiea is as gieat as in any other area. GocKlnian and Ward- 
liaugli addiess themselves to the problems of conducting reading- 
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lingiii.stic lescarth as each pciccixcs these piobleim. Theii appioathes 
are from \er) different |K"i.s|>ecli\e.s. and the leadei gains b) noting the 
dirrerence.s in thinking. 

On at least one point, l>oth Goodman and Wardhaugh are in 
agieement, and die) ai e joined b) Wanhop.a statistician, in stressing the 
necti to fmd appi opriatc tools foi conducting i ese.n ch and in i ccogni/.ing 
the dangers inhei ent in ovei i eliancc on statistically sophisticated designs. 

Longitudinal efforts 

Among the inoie promising geneial types of research is longitudi- 
nal. It is a type, lathei than a specific design. Blake and Allen discuss 
\aiious lesearch designs appiopiiatc foi use in conducting longitudinal 
studies. 

In the (iehl of reading thei e exists a pIctluM a of research of various 
kinds. That abundance does not exist when one attempts to identify 
longitudinal studies, which aic staice. If rcseaich tends to be a one-slioi 
activity and much of that is disseitation lesearch, pait of the problem is 
explained. Docioial students geneialK do not have the time to conduct 
longitudinal studies -at least not as doctoial piograms tend to be set up 
now. No doubt man) post<loctoial lescaicheis feel the piessuie of "pub- 
lish oi perish,'* and this piessuie does not peimit them the luxury of 
longitudinal research efforts. 

\Vev\i)ndei if Kjngitudinal research leallv is a luxui ) oi whether it 
is a necc^ssit). In so many otiiei fields. thc3-minute datatolleclion pime- 
duie that is so widelv pievalent in our fichl wouhl Ik* viewed with the 
gi a\ est alai m. II the individual collect ing the bi ief sample of data bapti/e.s 
them with a lil>eial sprinkling of holv watei statistius. they tend to Ik* 
aueptetlas bltssetl. If the statistical tiesigii thi/zles enough, we foigivethe 
fact that all soits of toiicliisions are diawn on the basis of one poorly 
designed test aibmnistercxl to X numlKi of children at one bi ief .sitting. 
.\nd al! .soits of in.tppiopi iate implications aie drawn fmiii diesc data. 

Apprt)piiateIongitudhial ie.seaiclicaiigivc us aiisweisio questions 
of majoi significance anti iiupoitanee. Peiluips it isoui (mly means to do 
so. The UM of leading Ix-gs foi caiefully designed, well-executed lon- 
gitudinal investigations, which h,ive Ikcu heiaUled in othei fields/rhev 
offei pi oini.sc of bi caking new giounds aiu! flev eloping new insights. 

Intcrclisciplinarv cl fbris 

lietause reading iiitei faces with so many other fields, it becomes 
important that wc kiuiw what thtiseothei disciplines have tooffcr. It is. of 
coui.se, iuipossible foi most of ustt) know a lutmlKi of fields in ascholailv 
sense. Iiicreasinglv, then, ii wouki seem that interdisciplinary research 
efforts should Ik' cncouniged. 
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Such clfoits <I«itc iMck .1 minilK'i of )c;ir.s .iiid sMiid oiil proiiii- 
iieiith bcc«iii!>c (ifilicii icl<ili\c .s|)«iisciic.n> in llic liicr.iliiic. Al Icmsi one 
such rcpui I , nnu over 30 \ c.n .s oUI, pi o\ itlcs one niotlel for iniei clibciplin- 
dry rese.ncli. We lefei lu Rol)inson > Why Pufnls Failm Reading (19-16). In 
tliis p«n'licul.n sludy, «i rcMthug peison served *i5 die tcHJidinaior .nid 
nuegr.ilor for die overall ie.se.ncli effort. If die focus is lo be rcMdiiig, .is 
we uuuh! Iiopeit ucniUI Ik, dieii die iiindel h .is \alid.iiid useful iiou as it 
was dieii. 

Iiilerdiscipliiiar\ efforts can pnnide us widi more diaii Jusi die 
insiglils offeietl In scliolais in allied tliutipliucs leladve lo dieii iiisiglils 
into reading and die leading piotess. Such effoiLsaie lo l>e encouraged 
hetause diey ina\ also ujkii padis to new iy|x;s of leseai ch approaches 
used in odiei disiiplines. The iiileicliaiige of i<leas. die coiiiparisoiis of 
I eadiiig as viewetl fi oiiisevci a! outside tliscipliiits. an<l die oppoi luinly lo 
lap knowledge bases lliat max be oiheiwise uii.i\.iilable lo us all make 
inieidiscipliiiai) cffniisaiiimpuiiaiillvpeof leseaich foi iiieiiibeisof die 
le.idiiig held to puisne. Befoie il can Ik: iiiideri.ikeii wiih an) degree of 
success. Iiowevci. we iiiusl leain lo coiiiimiiiicale wiih meiiil>ers of ulliei 
professions. 

Olhcr problems 

.\faii\ piol)leiiisof i«.ntluciiiig I cscaicliai e discussed in die \aiious 
papeis included in iliis \olmiie. .Some uf ihuse have alie.uly l>eeii dis* 
cussed: others need some ref erence at this |)oiiit. 

Bligh makes spcxifii iefeiciicestfitheiiiaii\ measui eiiieiit issues in 
rc.idiiig restaicli, Aiiioiig tliuse, he lefeis to the ciiterion tests used. As 
edil<iis of the Qttatttfl), wc have the o|)|Hirtuiiil\ to see scoies of maun- 
siiiptseacli veai . We aie const. iiitb ani.i/ed .it the c.ivaliei -like maiuiei in 
which tc\stsaie ticated. It is nut unusual to find no lefei eiices wliaLsoe\er 
lo the leliabilitv and validity of new oi autliui -developed iiisti uiiiesits. 
Kvensachlci is to hud .( w hole stodx based on the .idiiiinistiatioiiofse\eial 
tests, tliescoic sof winVhaie then con elated, factoi aiiaU/.ed.ui treated in 
soMicstatistic.diiiaiiiici. riic hiidiiigsaic then tie.ited as tint li without r;.*n 
ipiesiioii lH:iiig i.iised b\ the authoi relative to the appiopii.iteness oi 
\alidit\ of the iiistiuiiieiits useil. The aiithoi .issuiiies. and e\]K*cts his 
audience to assume, that thescoiesof the tctsts aie infallible and that the 
tests are assessments of ti nth. We in the fiehlof re.idiiig have leaclied all 
soils of conclusions and have made all Soit.s of decisions based on higliU 
sus]>ect instruments. 

Once the data are collected and analyzed, the researcher must 
interpret the meaning of his fnidings. Man is piesentsa inmilK*r of can- 
lions foi the loiisumer of icscuch. I lis cautious arr also applicable to tlie 
pnKiucer. If follow c'd, tliev would result in asu|)erior product. 
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Schools ha\c been less than warm in their leception toward re- 
search and researcheib — often for good reasons. The researcher has 
freqnentl) neglected to recognize that he sometimes disrnpts the class- 
room or school rontine in collecting his all-impurtant data. More than 
C)CC(isionall) he does not take the time to inform school personnel \\h) his 
project is worthwhile, but rathei .jssumes that everyone will be as im- 
pressed and excited about it as he is. An.btasiow presents many pi actical, 
helpful suggestions for the investigator who intends pursuing research in 
the school situation. His ca\ eats, if employed, tan onl) smooth the way to a 
happier experience on the part of both researcher and school. 

As Tone notes, a research report can ha\e only limited effect until 
it is shared with an audience. The final step in the good research report is 
pi esenting it in an organized, concise, and clear mainiei . More than one 
paper has been rejected by RRc)ad\isory board members because the 
author was unable to present his ideas in an understandable manner. 
Good oigani/ation and writing are difficult to teach. The Tone paper, 
applied, should go a long way toward aiding the writer in presenting his 
ideas well. 

This \oluine grew out of an obsci\ed need to improve research 
efforts in reading. We turned to mcmbeis of our adv isoiy board to write 
many of these papers because we f It they i epresented an expertise rarely 
found in one small group of indi\iduaLs. It seenied best in a first effort to 
focus on the neophyte reseaicher. Often that indi\idual is more open to 
recei\ing aid because he is awai e that he needs it, his ego is not as in\ol\ed 
as is that of the matm e l eseai cher. 

All problems of producing bettei research cainiot be sohed in one 
\oiuine. Howevei , it is hoped that the papers in this \ohiinedo help some 
researchers — who might otherwise ha\e tinned out another badly de- 
signed and much flawed study — to produce a product of which both they 
and the profession can be proud. 
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Identifying and planning, reading research 



HELKN M. ROBINSON. Emeriius 
Univenity of Chicago 



A potential reading researcher needs to be aware of a myriad of 
pi ubieins before he decides to undertake research in reading. Perhaps die 
major problem is to determine that there reall) is a field of reading quite 
apart from that ol related disciplines. Unless the researcher views reading 
as a whole dynamic area, he is likel) to study bits and pieces dial may ha\e 
little or no significance in furthering understanding of the re«iding field. 

Since the early 1960s, various models of the reading process have 
been developed in an effort to show how the different paits of reading fit 
together. None of these models have been widely accepted as com- 
preliensiv e, y et each attempts to show w hat is know n and w hat needs to Ix: 
learned, and especially how the parts may fit together. Without some 
overall view, research may be inconsequendal, 

Some reassm ance that there is a field of reading research may be 
offeicd by the fact that during the past 50 years, 8,230 .studies have been 
identified and suinniari/.ed in the "Siunmaries of Investigations Relauiig 
to Reading ' (VVeintraub et al.. 1974). The majority of the studies, regard- 
less of their scope or quality, weie done by persons who consider leading 
to be their major field of study. 

hiioughout this period, however, persons from other disciplines 
have approached the study of reading as though it were a special area of 
another discipline. F^or example* the linguist may reason that reading is a 
special type of language study, the psychologist may argue that learning 
lo read is an application of principles of learning. The sociologist may 
rightly see i eading as a reflection of social change and as an instrument for 
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producing that change. The librarian ma) \ leading ab a means of 
continuing self-education and/oi a ua) to sort out and establish human 
\alues.The facuhai reading einbi aces allof these\ieus,and man) tuhcrs, 
is important because, without such knowledge, man) factors are o\ei- 
looked in a single stud). Therefoie, the potential reading researcher 
should see the correlation of all lelated \ieus in extending the bod) of 
knowledge uhich is uimniel) reading. To the extent that it is possible to 
unite fragmented reseat ch fiom other disciplines and to bring it to bear 
on the stud) of reading, it becomes a discipline in its own right. 

Identification of a research problem, therefore, may require 
backgiound and imderstanding of related disciplines as u ell as of the field 
of reading. Stich understanditigs ma) be e\ en more essential in plaiming 
the reseaich because of the procedures and techniques among uhich the 
resea rebel must choose. The uide choices that nuts t be made in each stage 
of lesearch lead to the ' s)stematic imjuir) lor \ erified knou ledge" (VVi.se, 
Nt)rdlx:rg, and Reit/, 1967) which is the goal of all reading research. 

Identifying research problems 

A consistent I c .iew of research, )ear after year, for about 40 years, 
has shown a bioad gainiu of topics repotted. In preparing foi this paper, 
a ninnber of people uere asked, indiiecti), how the) chose their topics. 
The most conmion repi) was ^'because I am interested in it." This is an 
impoitant icason foi making a choice because withouta deep and per\ad- 
ing interest, few studies are cairied through the labored processes to 
completion. Interest, howe\er, ma) ha\e different origins and be of 
diffeient le\els of intensit). It ma) arise from a genuine cuiiosit) toleain 
something which is not known. Interest ma) be generated exteinall) b) 
piessures, one of which is often called the need to fmblhh. In this case the 
interest may be much less intense than in the case of curiosity. 

Othei researchers are those who produce doctoral dissertations. 
Doctoial students also \ai) gieatl) in the intensit) of their interests in 
gi\en topics. One student ma) be so much inteiested in a topic that he will 
spend exce.ssi\c time to de\elop means foi stud) ing the problem. In 
contiast, anothei student ma) select a topic which can be easil) studied, 
with his major interest being in completing the dissertation. 

While there is a great varietv of reasons foi choosing topics for 
icse.ncli, the most common ones can be grouped into se\eial categories. 
The lategoiies \ai) from accidentall) finding data that ma) produce a 
publishable stud) to using the i|uestions geneiated b) one stud) as topics 
for succeeding ones. 

1. Seeing available data 

In this categoi) aie the post hoc studies in which the researcher 
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finds data w liioli have not bccii pi ocessed or reported. In most instances, 
no advance plan has been made to collect the data. In selecting such a 
topic, the researcher shoidd be aware of the problems he may face when 
his study is completed. 

Examples of this l> pe of choice of a topic can be found in compara- 
tive studies of reading achievement of schoolsvof chisses, of methods of 
teaching, or of school or classiot^m organization. Reading achievement 
test data may be found for fourth grade pupils among whom some of the 
children w ere taught h) method A and others by method B in first grade. 
If the researcher chooses to use such a topic, he must recognize the 
confounding factors that ma) have occurred during and since first grade 
which will reduce the validity of his stud). He may not even be able to find 
outliov\ the tests were given oi scored or whether he can depend on diem 
to hav e measured an) desired changes. He must be prepared to doubt any 
of his findings and be uncert.iin of interpret.itions. 

Ill contrast, data may be found which were carefully collected in a 
study for aiiotliei purpose but could be analyzed equally well for a new 
purpose. I f the researcher has access to all of the details of the first study 
and finds dial the) meet the needs for his prospective study, then he may 
anticipate results as dependable as those of the stud) for which the data 
were originally collected. 

Topics for research chosen because data are found, without ad- 
vance planning, are not plentiful and, therefore, the researcher should 
approach such a choice with great care. 



2. Seeking practical answers 

This category includes the identification of a topic w hich will help 
school personnel solve problems. This category differs from the preced- 
ing one in that the stud) may be planned in advance. In some instances, 
howevei, school administrators simply request evaluation of the effec- 
tiveness of certain plans and procedures. The researcher who chooses a 
topic in w hich all data except current evaluations are in retrospect must 
anticipate questionable lesults. For example, such factors as the types of 
pupils, the qualit) of instruction and enthusiasm of teachers, the amount 
of time given to reading instruction, and the facilities available may not 
have been recorded over the years of instruction. 

The researcher who chooses to help school personnel solve prob- 
lems of instruction may do so b) insisting that the plans be made in 
adv ance and that adequate procedures be followed. Thus the researcher's 
topic is more likely to satisfy him, as well as help the particular school solve 
problems, without the bias and confounding effects often found in re- 
ports of research* The researcher w ho chooses topics of this type needs to 
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be aware that the i esuh.s may be limited to the school s) stem with which he 
is dealing. 

3, Searching for a problem 

Many graduate students, as well as others, may search the literature 
for unsolved problems w hich appeal to them. There are common ways in 
which the literature is used to suggest topics to those who seek problems. 
First, the individual may read in an area ofinterest to him and locate one 
or several problems that he would like to study. Of course, he must make a 
thorough search of the existing literature on the topics to find out if these 
topics have Ixren studied previously, what answers have been found, and 
whether there are still problems to be solved. 

Second, some graduate students who have not had an ade(|uate 
foundation in existing research look for sources of unsolved problems 
publi,shed by researchers from time to time (Durrell, 193G, Gray, 1952; 
Robinson, 1968, 1970), 

Third, the person who has systematically covered the research of 
the 1900s should identify so many problems that need solution through 
research that a choice of the many topics may be difficult. 1 n addition, one 
who has such a background w ill usually keep abreast of ongoing research 
reviewed annu.illy in Reading Research Qiiarterly, in the unmiiil Proceed- 
ings of the National Reading Conference, and in Elementary English, The 
problems of choosing a topic for research are quite different for this 
person than for those uho have had little contact with the literature and 
have limited familiarity uith unsolved problems in reading because so 
many problems become evident ttiat he may not be able to decide which 
one to choose. 



4. Obtaining guidance 

Doctoral students especially, but also some neophyte researchers, 
may ask established researchers for suggested topics. Sometimes they 
send a short questionnaire asking the mature researcher to suggest im- 
portant topics; from the ideas secured, the students may select the most 
appealing ones. In addition, such a questionnaire may ask about topics 
which would be investigated if unlimited time were available to the estab- 
lished researchei , A variety of other (|uestions may be included to provide 
leads to topics considered important. 

The doctoral candidate must satisfy a major professor or advisor, 
and usually a committee, that his topic is significant. Professors vary 
markedly in theii pr(Kedures in dealing with doctoral students in this 
respect. Some assign topics closely related to their own interests or to 
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expand their own research. In contrast, other professors insist that stu- 
dents identify a topic in which they are interested, even though it may be 
totally unrelated to their professor's interest. 

The doctoral seminar at some universities provides opportunities 
for those studenLs seeking a topic for research to exchange ideas with 
others who are gathering data or have completed their studies. The 
advantages are numerous but two deserve mention. One is that the 
problems inherent in research on a given topic may become evident and 
either discourageor encourage the selection oftopics likely to pose similar 
problems, A second advantage is that the beginner may see aspects of the 
problem which have not been studied, or extensions which need to be 
undertaken. Techniques for studying a particular problem may be seen as 
means of exploring a topic in which the beginner is already interested. 
The seminar may save time for graduate students wishing to identify a 
topic for research. 

5. Enlisting in a cooperative project 

In some instances two or more graduate or postgraduate students 
wish to select a single topic and carry on cooperative studies. When 
cooperative research is planned by a mature researcher and each student 
carries out one phase of the study , the student may complete his disserta- 
tion morequickly ; but he should bcaware of the experiences he is missing. 
Specifically, the student does not learn, under guidance, how to identify a 
problem and plan it himself. 

How ev er , cooperative research may be a valuable experience when 
two or more persons with differing backgrounds plan to attack a problem 
from different points of view. New insights may be extended while the 
education of those who cooperate is broadened. For example, a person 
with a major background in psychology or sociology who teams up wiUi 
one whose background has been in teaching may bioaden the research 
base of the topic they choose. Although persons from various disciplines 
plan together, each should be responsible for developing his own aspect 
of the study so that he may learn how to become an independent re- 
searcher. 

Graduate students w ho have an opportunity to work as assistants to 
professors on their own studies may learn a great deal if they are able to 
participate in the early phases of choosing topics and planning research. 
In addition, after the professoi's problem is delimited, numerous ancil- 
lary problems may be obvious so that they can serve as topics for study by 
students. 

The Reading Research Center at the Univ ersity of Chicago, with its 
multidisciplinary advisory committee, provided opportunities for 
graduate and postgiaduatc assistants to Icai n about identifying and plan- 
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ning research. Uiidei dose bupervision, beginners did routine tasks, such 
as giving and scoring tests, carrying out observations, and searching the 
literature on given topics. As students became more competent, they 
accepted greater responsibility . Postdoctoral research associates were able 
to codirect a studv and/or de\ elop one of their ow n. All staff members met 
to discuss problems as they arose, and the Advisory Committee brought 
the knowledge of several disciplines to bear on research problems of both 
the staff and the students. 

Neophyte researchers who wish to have experience with mature 
researchers may become members of research and development centers; 
some of these are independent while others are located at universities. 

An example of an earlier extensive coopei^ative project involved 27 
centers (Bond, 1966) with numerous experienced and less mature re- 
searchers. First grade methods of teaching reading were being examined 
and, in most instances, the choice of topics was narrowed to two or three. 
Laige coopei ati\e studies of this type are rare, but they can be a source of 
tiaining and experience, both in identifying topics and in planning and 
conducting research. 

6. Continuing to study a topic 

Those persons who use this technicjue for identifying research 
topics are limited in number. Pi obleiiis in reading will never be solved by 
"one-shot" suidies, as Barton and Wilder (1964) characterized them. 
While single suidies ofa given topic may be useful if they entera newarea, 
if they develop a promising technicjue for inv estigation, or even if they 
show that certain hypotheses are not tenable, single suidies usually just 
scratch the surface. Indeed, most of them open up more problems than 
they solve. For example, a good doctoral dissertation shoukf suggest 
enough research topics to launch a researcher on his major career if he 
continues to pursue these unsolved problems. Unfortunately, few people 
in reading follou one study after another on the same topic until some 
dependable answers are obtained. Perhaps ihis failure accounts for the 
characterization of reading research as fragmentary and of less value than 
it should be in view of the thousands of reports available. 

There are tremendous advantages in continuing to investigate die 
same topic, if it is significant to the field. The researcher already has a 
background of previous investigations, has developed a rationale for the 
topic, and most iinpoi lant, has had first-hand experience in using promis- 
ing technicjues so that he recognizes limitations which he can plan to 
eliminate. 

Another aspect of this category involves other persons continuing 
the investigations done by one researcher. As a result of an ongoing 
doctoral seminar at the University of Chicago in which each student 
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reported on his research peiiodicall) up to the final oral examinations, a 
series of studies was done on \isual and auditory perception. Uitcr the 
writer combined the results of these studies and did a larger investigation 
than could be expected of any graduate student (Robinson, 1972). 

In the past, most continuing studies have resulted from a new 
technique for investigation. No doubt many examples are familiar but two 
or three may be useful to neophytes. The development of the eye- 
movement camera and its impro\ ed successors sparked do/,ens of investi- 
gations of how the eye behaves during reading. Studies spanned the 
gamut of age from beginners to adults and of content from prose and 
poetry to pictures; and it probed behavior in a number of different 
languages as in the Unesco report (Gray , 1963). With refined methods of 
recording and computer counting, the technique is currently in use to 
determine time for processing input \ersus time for saccadic fixations 
( Abrams and Zuber, 1972). Along u ith the inq^rovements in instrumenta- 
tion has come greater sophistication in research design and inq^roved 
statistical techni(|ues to determine the dependability of the data secured. 

A second example of continuing to use a technique is current. The 
cio/e procedure was developed by Taylor (1954) and has been u5ed by 
do/eiis of researchers uiih different populations and for different pur- 
poses (Rankin, l%5 and Jongsma, 1971). Concurrently, some research- 
ers such as John Bormuth are continuously exploring the clo/e procedure 
asa measure of readability and as a measure of comprehension; and they 
have related it to various linguistic units. 

I here are, then, many and varied ways used by prospective re- 
searchers to identify topics. Neophytes may need guidance while mature 
researchers u.sually have the next topic in mind before the preceding 
study is completed. 

Choosing among topics 

When the prospective researcher has found several oi a multitude 
of problems begging for solution, he cannot proceed on all topics at one 
lime so he must choose the one he will pursue fir,st. Thi.s choice involves 
move than interest in a topic; it in\ oKes the competency of the researcher 
in different areas and available facilities to do the study. One way to 
examine the practical possibilities of conducting research on a given topic 
IS to look at the major areas into which past research has been classified. 

Since 1925 when Gray published his first summary, rescairh in 
reacling has been classified as the sociology of reading, the f)li\siology of 
reading, the p^yholog) of reading, .mA the teaihing of reading. (See the Annual 
Summary of Research Relating to Reading in the Reading Research 
Quartetly.) 
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The s(fciobgy oj reading includes topics dealing with how rea(Hiig 
fiiiKtions in iiidi\ idiial and social life so diat society can attain its goals and 
standards. It deals primarily with adults as subjects. One example of 
problems in this area is the extent of illiteracy and its relations to social 
problems, cmplwyment, nan(>»^'d de\elopment and the like. Others .ire 
reading in relation to other mass media; readership of newspapers, 
magazines, and books, and reading interests of adults. Readability of 
adult materials and the effects of reading are further examples. Cukural 
and ethnic factors as they relate to reading in the social setting are 
problems of current concern. 

Topics included in physiology of reading include the physical, 
neurological, and psychophysical factors in\ol\ed in reading. Studies in 
this area have included such topics as endocrine balance, vision and 
hearing problems, neurological disorders, effects of dominant and reces- 
sive hemispheres of the brain, and electroencephalographic results. 
Psychophysical factors ha\e included studies of the heart rate, respiration 
rate, and the like, especially in relation to what is read. 

Examples of topics classified as ihi: ftsychology of reading are experi- 
ments in learning to read, \isual and auditory perception, language 
abilities basic to reading, personality and self-concept, and readability. 
Causes of readingdisability or factors related to itco\er both physical and 
psychological categories. 

The teaching of reading includes topics dealing with methods of 
instruction and with materials, ranging from the earliest years to adult- 
hood. Also studies considering treatment of reading disability are placed 
in this category. 

Examination of research completed in these categories reveals that 
there are differences in methods of study, of instruments used, and of 
sampling— all of w hich may influence the selection of a topic for research. 
An example of a topic from each cat-jgory should illustrate some impor- 
tant differences. 

In the sociological area, a topic might be concerned with what 
adidts read in the United States. OIniously it\\ould be impossible to study 
all adults, so it is necessary to know how sampling procedures are de- 
veloped and used, to ha\e the procedures for securing information on 
what is read, and to have the background to interpret the data gathered. 

In the area of psychology , a topic could relate to visual information 
processing. To make such a study might require an instrument as compli- 
cated as a cathode ray oscillograph controlled by a computer. Moreover, 
the stimuli would need to be designed to be appropriate to solve the 
problem. The age and reading competence of the subjects would 
influence findings, and the problem of securing a representative sample 
of subjects should be considered. 
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Topics dealing with the teaching of reading are sufficiently famil- 
iar so thai illustrations are supei fluous. Usually the) are done in schools, 
although a few are carried om in laborcUories. A thorough discussion of 
this type of research appears in the Handbook of Research on Teaching 



In selecting a research topic, each individual should assess his own 
background to undernke the problem. For example, historical research, 
which is a $ur\e) of past events in reading related to the social setting of 
the particular period of time, can be done most readily by one who has 
competence in the general methods of historical research. Studies in the 
sociology of reading require an understanding of sociology and the re- 
search methods used b> sociologists. Unless the researcher is willing to 
invest considerable lime in developing a background in a particular disci- 
pline, he ma) wish to consider topics in which his background permits an 
understanding of research procedures and interpretation of results. 

Nciihei identification noi selection of a research topic in reading is 
a simple prcKess if an excellent stud) is anticipated. Man) factors must be 
considered before the plaiming of the study begins. 

Planning reading research 

Planning research is the most important step of the entire study. It 
is time-consuming and, if properl) done, ma) e\en discourage those not 
really committed to research. 

For doctoral students, discriminative selection of a topic and care- 
ful advance planning pa) large dividends in the qualit) of the final 
pnxluct and often reduce the casualties on the final oral examination. 

A number of steps must be followed in prep.iring a carefully 
conceived plan for research. 'Fhe most important ones are discussed 
briefl) , but the oi der ma) be changed, depending on prev ious in\ esiiga- 
tion of the topic and experience of the researcher. 

Searching the litci*aturc 

The previous literature search to idcntif) a topic has usually un- 
covered a great m.ui) sources .md studies which were read for that 
purpose. At this point, it is esseniicd to do careful docinneniaiy research, 
which is anal)sis of what has been done and how it was accomplished in 
relation to a specific topic. This surve) should be extensive and the 
analysis should beci iticak In other words, all studies related to the chosen 
topic should be evaluated to determine their strengths and weaknesses 
and their contributions to the solution of the problems. 

A half do/en or more checklists may be found to guide the evalua- 
tion of previous rcsecuxh, but guided practice appears to be the best 
means for acquiring the abilit) to read research criticall). As the studies 
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are examined, time is haved if full i efereiices and notes are made for each 
one. Some of these references will be needed later in developing the 
rationale and providing the theoretical framework for the report. 

' When the literature search has been completed, it is possible to 
determine the aspects of the chosen topic \\hich ha\e been successfully 
studied and those in need of further research. In addition, the methods 
for research can Ix; reviewed to ascertain the ones which were most 
productive and those which were least satisfactory. It is always possible 
that newer techniques of in\estigation or of data treatment have de- 
veloped and are applicable too. 

Delimiting die problems 

Doctoral candidates and inexperienced researchers often need 
guidance in delimiting their problems to manageable size because they 
tend to seek answers to too many questions in one study. On the other 
hand, the problems must not be so limited as to be trivial. If a list of 
unanswered questions is made during the literature search, it is possible to 
choose the most significant or pervasive ones to pursue. Often several 
questions can be combined into one that throws light on answers to the 
others. Sometimes it is essential to consider several components of a 
problem because each influences the other. 

Developing hypotheses 

In planning research, one of the essential steps is to state the 
hypotheses to be tested. Surprisingly, e\en some so-called experimental 
studies contain no slated hypotheses, and often no clearly slated ques- 
uons. Such studies may be compared to a "fishing trip" in which one is 
satisfied with any fish that bites the bait. Hypotheses usually state the 
expectations of the researchei ,//'the anticipated procedures are followed. 
The exception is the null hypotheses, staled primarily to satisfy statistical 
criteria. 

The researcher must be certain that the hypotheses are stated 
in a form that is testable. For example, if a hypothesis states that an 
experimental group will read better than a control group after a given 
treatment, what is meant? The words read better may imply oral reading, 
silent reading, or a higher (|uality of materials read. And how much is 
better— a point or two on a test, or a statistically significant difference 
favoring the experimental ^Toup? In other words, the hypotheses must be 
stated so that they can be confirmed or rejected. 

AnoUier problem in staling hypotheses is the implication of cause- 
and-effeci when the ([uestioii is really whether two changes occur 
together. 
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rheclaiil) aucl objccli\iiy of ihcMatcmentofhypolhescsguiclclhe 
pnxreclurcs the rcscarchei plaiii to folluw. M)poihescs buggcsi to the 
researcher the data uhich will be rclc\ani and enable him to reject the 
collection of irrelc\ant data. In addition, hypotheses project the 
Framework foi conclusions to be reached when the stud) is completed. 

Finally, the number of h)potheses generated may show the re- 
searcher that his stud) is too broad and not manageable. In turn, he ma) 
redefine his problem, lea\ ing less important h) potheses foi another stud) 
if necessnr) . Also, it is possible that the problem has been defined in such a 
way that no hypotheses can be tested b) known means. In this case, 
refinenienl of the topic may be in order. 

Estimating projected research time 

When the h) potheses are clearl) written, the researcher can esti- 
mate the time that will be needed to do the study. Usually even the 
CKperieiKcd researcher underestimates the time re(|uired to complete a 
study because so man) unforeseen problems arise. However, the 
nc{)phyte often has no basis foi estimate and, therefore, plans his research 
entirely innealistically. 

Two aspects of the time problem should be estimated. One relates 
to the number of man-da) s expected to be a\ailable foi the study and 
whether thev w ill be a\ ailable w hen the) are needed most. For e.xample, a 
study of certain characteristics of beginnei s in school may recpiire a great 
deal of time at the start of the school )ear. If time is not a\ailable then, 
changes in pupils will occur before they are studied. 

A second important aspect of time is the number of )ears antici- 
pated to test the li)potheses. For e.vample, some in\estigations of the 
teaching of reading must be extended se\ei al )eai s to determine w hether 
ad vantages oi disadvantages to a treatment are tempoiai) or permanent. 

A disproportionate or unreasonable amount of time doing a doc- 
toral dimMtation can be frustiating, lead to dropouts, and discourage 
future research. On the othei hand, a limited amount of time should 
ne\er take precedence o\ei the selection of a significant topic. Sometimes 
ancillary data can l eadil) be obtained to pio\ide a base foi a second study 
after the dissertation, but a realistic assessment of time needed foi re- 
search should be stressed In e\er) ad\isoi of doctoral candidates. 

Acquiring special competencies 

In planning research, it is important to a.ssess the special com- 
petencies that will be recjuired to test the hypotheses. Understandings in 
such areas should be developed before the study is begun rather than at 
various points duiing the in\estigation. Experimental studies require 
competence in measurement oi assessment and knowledge of statistical 



fdnitifyug and plarmmg research 

24 



anal)sij» ofdcitd. I liesc (.oiiipclciiucs would be aaiiiircd before planning 
the study because, as Wise, Nordlxrig, and Reil/. (1967) said, "Elaborate 
statistics cannot substitute for basic misconceptions, or for failures of 
experimental control . . . /' 

Statistics is an area in uhich many docioral candidates need to 
develop competence. They expect to gather data and compare the 
findings. As Cuba (1963) has explained, statistical procedures are not a 
research plan but a tool to provide inferential conclusions from proper 
samples of subjects. However, statistical procedures have developed so 
rapidly with the advent of computers jhatitis usually helpful to consult a 
statistician, no matter hov\ well-prepared the researcher is in the field. 
Consultation should (kcui during the planning stage to avoid errors that 
cannot be corrected after the data cu e gathered, l^he researcher who is 
competent in statistics c,»m be sure that the guidance giv eahelps him plan 
to test his hypotheses rather than to do a major statistical study. 

Competence in measuringor assessing veadingcharacteristicsto be 
studied by standardized tests may be acx|uired from <i critical review of 
similai investigations, from reliable sources such as Buros' Mental Men- 
sitrmetils Yemhuuk or from recent books aboul measurement in reading 
such as that by Fan (1969). However, many hypotheses are tested more 
piecisely by means othei than standardized tests. For example, in a study 
of fust graders attitudes tow aid leading w hen i.t.a. was used, an atlitude 
test had to be devised, tried out, revised, and finally used. Logs of time 
spent in teaching may Ix: needed. Observation schedules which permit a 
high degree of agieement among observers may have to be devised, A 
t|uestionnaiie or stiuctured inteiview letjuiies the same care in prepara- 
tion as any other instrument for a study. 

In addition to measurement or assessment of reading, it may be 
essential to become familial with ways to determine factors which con- 
found the experimental data. Many of these factors are not determinable 
by standardized tests. 

Securing sources of data 

Different kinds of sources are essential to different types of studies. 
For example, historical research is based on original documents. Com- 
parative research can Ix; done only if subjects with different characteris- 
tics are available to compare. 

Expei imcntal research in reading usually is done with children but 
sometimes vvith adult subjects. In planning research, therefore, the 
hypotheses should guide the investigatoi to a specification of the charac- 
teristics of the subjects needed, age or grade level; good, av erage, or poor 
readers or an nnsclectecl gioup; high or low socioeconomic levels. 

Schools are the inajoi source of subjects, so the researcher should 
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identify sthoolb eiuolliug children appropriate for collecting data. Then 
it is essential to obtain cooperation from the schools. Usually permission 
comes from administrators, but sometimes from school bo.nds and par- 
ents too. Teachers are extreniely important in cooperative efforts in 
research. If children aie to come to a laboratory, more detailed permis- 
sions and cautions are needed than if they are to be studied in the school 
seuing. It is w ise to secure \v rilten agi eements between the researcher and 
the schools. 

If appropriate subjects cannot be obtained, the problem may need 
-to be altered oi e\en abandoned temporarily. Unselected groups of adult 
subjects are e\en more difficult to obtain, college students are frequently 
used but they represent a selected group. 

If a study is planned to re(|uire a short period of lime, the rate of 
turno\er of subjects is not so crucial as when the study is planned to use 
the same subjects for se\eral years. In the lauer case, a search should be 
made for schools with stable populations or school systems willing to 
permit researcheis to follow subjects, at least those who transfer within 
the system. K\en so, one should plan to include alternate subjects toa\oid 
excessive potential losses from year to year. 

Doing pilot studies 

One w ay to ascertain the le\ el of coopei ation of a school sy stem or a 
particular school is to carry on a ministudy. Foi example, the writer 
selected an innei city school that made e\ ery adjustment needed to gather 
data easily. At the same thne, a saburban school, known for its interest in 
reseaich. re([uiie(l a yeai of prepaiation before the experiment could 
begin. The pilot study permitted the i esearchei to check the adecjuacy of 
randonj assignment to classes before school opened, enabled the re- 
searcher to meet with parent groups in ad\ance, and assured smooth 
operation of testing and c)bser\ ation early the following school year. 

Pilot studies serve other purposes and one or seveial may be 
needed. They help the expeiiinenter refuie his techniques lor in\estiga- 
lion, try out his data-colleuion schedule, and gel acquainted with the 
teachers so they do not feel apart from the experiment. 

In unexplored areas, a mmibei of pilot studies may be needed. For 
example, Piekaix (1954) cairied on a do/en oi so pilot studies to identify 
the best selections foi her purpose, and to find the most productive 
sectioning of the selections so that the gieatest amount of \erbali/ation 
about retrospective thinking could be obtained. 

Pilot studies often lead the researcher back to refinement of his 
hypotheses and/ui plans foi data gathering which lesult in impio\enient 
in the main experiment. 
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A,pilot sludy shoiilcl never be confused with a major study. Seldom 
are these tryout studies publishable unless the technicjues used in them 
might save the time of other researchers. 

Writing the plan 

Each research phui should be written, even in its earliest and 
crudest form. As a rule, each plan needs to be revised and sharpened 
many times during the period of planning. When it is in final form, the 
written plan commits the researcher to each step of his procedure up to 
the completion of the study. 

As the plan is written, the researcher should be especially alert to 
any assunipuons he has made and, wherever possible, the number should 
be reduced. It is unwise to make an assumption if there is a way to avoid it. 

In planning research, a few limitations may be unavoidable, but 
notations should be made in the plan. F'urther limitations usually appear 
during the course of data collection. The prospective researcher needs to 
ask. in advance, wheUier there are too many limitations to interpret any 
>findings with confidence. Then he can alter his plan or approach die 
problem from another vantage point if necessary. 

The final part of the plan is speculation on the various alternatives 
that could result from the findings. For example, if one hypothesis is 
supported and a second one rejected, the researcher should anticipate 
how this could occur and what coiild account for it. By examining the 
meanings of the possible outcomes, the researcher is anticipating rela- 
tionships among the hypotheses, that is, he is reasoning about whether 
each supports or refutes the others as they are written. The projection 
of alternative outcomes may lead him back to reexamine his hypotheses 
or it may suggest that he is prepared for any results that are found. 

The carefully chosen topic which is planned in accordance with 
accepted and thoughtful procedures is likely to make the greatest con- 
tribution to knowledge about reading. Improved research planning can 
contribute immeasurably to the quality of reading research. 

REFKRKNCES 

ABRAMS. S. O., & /.UIJLR, B. I- Sonic icimHiral LlwraacnsliLs of information pnKcssnig during 

TcatWu^, nradmg Rfuarch Qiartfrly, Kill 1972, 5, -10-51. 
BARION» AI.I.KN M.. & WII.DKR. I)A\ ID K. Research jnd practice in the icathing ofrcaclijig a 

progress reiH>rt. In M.uthew B. Miles (E<!.>, Intimvitiom m fdiuation. New York. Icaclicrs College. 

Coinnihia Univcrsii>\ lOG-l. 
B LjROS. OSCAR k. (Kd.) A/^-k^/ mfosurrmftibyarM. HigliUntI l\irk. Ncu Jersc> Gryphon Press. 1938 

and inieraiUiently thereafter. , 
CAMPBELL. DONAI.I) L.^SfANLEV Jl^LIAN C. Exi>erimemal and musicxperiinemal designs for 

research on teaching. In N. L. Cage (E<Lj. Handbook ofreseanh on trmhing Chicago Rand NlcNally. 

1963. Pp. ni-LMG, 

t)URREI.L. DONALD D. Research prt;l)leins tn the elemeiuary !>dllK^s.t.IrtnenU^^y English Rntcu, April 
imjh H9.l56and May 1936./?, I8-M92. 




ROBINSON 21 

27 



^ '^^^KCXJFR Hmlmg, what i an (ff mruiuml? Ncv\.uk, IXiu\.irc. liitcriiatnMi.i! KcMdiiig Associaiioii, 

CRAY, WILLIAMS. Nmletl rcscjitli in tvMhti^J.trmentan hnglish, Fcbnur) 1952,29, lOO-IOO. 
GRA Y, \V1LLIAM S The teaihmg of remhng atui a'ntmg. Pans. Uncsto, !y()3. (Distributed !>> Stolt, 
rorcsinan, Glcnvicw, Illinois) 

GRAY. WILLIAM S Suininarjr of iinestigations relating to reading. Supplemftttary tducatwml motuh 

graphs. No. 28. Chicago: University of Cliicago Press, 1025. 
(JL'BA, KCON' C. Cuininon aciinc about experimental design in cdiuational research. UiijMiblished 

paper read .It a Faculty S>iiij)osmin, School of K<luc.ition, New \ork l>iii\ersti>, Fehriur) 25, 1963. 

JON'GS.MA, EL^GENE ihr doie {mHedure ma Uathingtahntqiw, Newark, Del.m arc. International Read- 
ing As^jciatioii, I97L 

riKK\RZ,JOSKriIINK Individual differences in inici pretaiion. Ln!)iiblished diKtoral dissertation. 
University of Chicago, I95L 

^ ^*^W5!^'V5'^^33 H?"^*^ prixetlurc a sur\e> t>f research. \ mrbovk oj the South M o/ HnuUng (.onjfrtme, 

ROBIN'SOV, HELEN M The ncxuleiadc In Helen M. Robinson KhiU,lnnovatiunitndihangem remUng 
imtnutton. Sixty-Seventh YearlK>ok of the National Swiciv for the Study of Education, Fart if. 
Chicago; University of Chit^go Press, 1968. Pp. •*0<).-J20. 

ROBIN'SON. MEI EN M Significant unst^Ived ptobleinsiii UAihue. Journal ut He<uimp, Novciiilxir 1970, 
77.82 and I:M. in. hJ J h 

ROBINSON, IIELE.N M Visual and auditor) nuMlahites related to niethwls Un beiiinninK readiii}!, 
liemUng Research Quarterly, 8, Fall 1 972, 7-11. ft 6 

'^^\'''i9M'-\y*''^*^^'^ ^ priKedure a new t<H>l U>r inoasuiing rcM<labiIit),yo«r7w/um Qtmrterh, 30, 
Fall 1953, 

WEIN I RAUB.SAMLEL. ROBINSON, HELEN .M , SMI I II. HELEN K., PI.F^SAS.CUS, & ROWUS, 
MKIIIAEL Suiniiiar) of nncsligations relanng to rc.iding, Jul> I, 1973 tojtmc 30, 197-L Heatlmg 
Research Qiiarterly, 10, 3. entire issue. 

WISE. JOHN E .NORBERG. ROBERT B.. & REIT/.. DONALD J. Meth^nU vf researth m education. 
Boston; IX C. I Ie;uh, I JK>7. 



22 Identifying and planning research 



ERIC 



28 



Design problem in reading research 
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It is not easy to do nieaningful research. It is no easier in reading 
than in any other field. First of all> good research depends on asking good 
questions. In this discussion of research design, it is assumed that you 
already have a good quesdon without even worrying about what criteria 
are applied to decide the "goodness" of the question. 

Most good questions deal with one or more of perhaps four major 
categories related to the reading process: theoretical formulations, in- 
strucuonal techniques, learner skills, and what might be called *'mea- 
surement research," For example, research related to theoretical foririu- 
lations focuses on such orientations as psycholinguisUcs or information 
processing; research on i nstructional techniques mightconsider a phonics 
approach, the Distar program, or some other special training program; 
research on learner skills emphasizes such issues as one versus several 
"ability factors," the relationship of reading to listening skills, or differ- 
ences among various subgroups of learners (so-called "disabled readers*' 
are a popular subgroup for research), and measin ement research deals 
with such issues as how to measure comprehension or problems in diag- 
nostic assessment. 

Superimpose on this categorization a distinction between explora- 
tory and confmnatory research. In the former, one adopts a stjmce of 
"ignorance" about some phenomenon and designs a study which will 
hopefully reduce the level of ignorance and e\en suggest questions for 
fiu ther study in confiimator) research. Confirmatory research involves 
the formulation of specific hypotheses (informed guesses, predicdons) 
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alxnit a phenomenon, followed by the design of a stud) which becks to 
confirm — or disconfirm — the guesses. 

A crucial component in the de\elopment of a research design is a 
clear and logicaljustification for the predictions one makes. This is proba- 
bly the most neglected aspect of the research prcxess. We are prone to 
pKxreed as if our hypotheses ha\e sprung full-bloom and are so obvious as 
to need no justification. Again, in this discussion it is assumed diat you 
hdveclearl) justified the hypotheses you are to te.stalthough much of what 
follows is related to this issue. 

In reading research, the predominant modes of inquiry require 
the collection of information, usually quantitati\e information. The col- 
lecting of such information is usually mediated by various tests, question- 
naires, or observation instruments. Many an otherwise worthwhile re- 
search study has foundered on measurement inadequacies. Indeed, the 
measurement problems may be the most important limitation to "good" 
reading reseaich. Since these pioblems are discussed by Bligh elsewhere 
in this volume, they are not treated here. 

Having ruled out question asking, hypothesis justification, and 
measurement problems, what is left for us to consider? What is left is the 
problem of developing a research design, which in fact answers the 
{|ucstioii or (|uestions asked. A good research design is one w Inch accom- 
plishes two majoi purposes: it allow.s us to rule out alternative explana- 
tions for the phenomena we observe, and it leads to mininii/ang errors — 
both inferential and stadstical. 

Research design to rule out alternative explanations^ 

One of the best guides to doing **good" research is a 1964 article by 
John Piatt in Science magazine, "Strong Inference." Although it was writ- 
ten from a physical-science perspective, it seems especially relevant for 
educational researchers. A systematic application of the philosophy ar- 
ticulated by Piatt would yield more significant advances in our under- 
standing of reading in the next decade than in the last half century . That is 
a strong statement— especially w hen it turns out that w hat Piatt advocates 
is es.sentially what we have all been taught to call "the" .scientific method. 
But let me paiaphrase parts of Platts article before you pass judgment. 
He asks: 

Why should iheie be . . . rapici advance in some fields and not in 
others? I diink the usual explanations tliai we tend to think of— such 
as the n actability of the subject, or die (|uality oi education of [those] 



r^o cxircincly useful rcfcreiitcs here .uc Campbell mm\ Stanley's monograph (1966) on t^penmrntal 
ami Qitau Expmmmtal Dfuj^LK «tiul .i IxmA by Daviil Cox ( 195S^, nanntngo/t^prnmrnts. lAcu together, 
these sources provide inval(ia!)!e guidance (or the researcher. 
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drawn into it, oi the si/e of research contracts— are important but 
inadequate. I lia\e begun to Iwlieve tliat the primary factor hi scien- 
tific advance is an intellectual one. Hiese rapidly moving fields are 
fields where a particular method of doing . . . research is systematically 
used and taught, an accumulative method of inducti\ e inference that 
is so effectiv-e that I think it should be given the name of "strong 
inference" tp- '^'^7]. 

Strong inference consists of applying the following steps to every 
[research] problem . . . , formally and explicitly and regularly: 

1 . Devising alternative hypotheses; 

2. Devising a crucial experiment (or several of them), w ith alter- 
native possible outcomes, each of wiiicli will, as nearly as possible^ 
exclude one or more of the hypotheses; 

3. Carrying out the experiment so as to get a clearresult; 

4. RecycHng the procedure, making subhypotheses or secpiential 
hypotheses to refine the possibilities that remain; and so on. 
It is like climbing a tree. At the first fork, we choose—or, in this 

caser, "nature * or the experimental outcome chooses— to go to the 
right bianc b oi ihe left; at the next branch, to go left or right; and so 
on .... 

On any new problem, of course, inductive inference is not as 
simple and certain as deduction, iKcause it imolves reaching out into 
the unknown. Steps I and 2 [devising hypotheses and experiiiieiitsj 
require intellectual inventions, which must be cleverly chosen so that 
hypothesis, experiment, outcome, and exclusion will be related in a 

rigorous s) llogism What the formal schema reminds us to do is to 

11) to make these inventions, to take the next step, to proceed to the 
next fork, uitliout daudliiig or getting tied up in irrelevancies [pp. 
347-348]. 

The strong-inference attitude is evident just in the style and 
language in which the papers are written [p. 348]. 

Such papci s arc chai actcri/cci by the inclusion of buch statements as ''Our 
conclusions . . . might be invalid if . . . (i) . . . (ii) ... or (iii) ... [is true]. We 
shall describe experiments which eliminate these alternatives." Or, in a 
discussion of competing theoretical explanations, we find a list of proposi- 
tions derived from those theories u hich are "subject to denial"; the author 
also suggests which ones would be "most vulnerable to experimental test." 

Such a research attitude is almost the inverse of the kind of think- 
ing wliicli characterizes most reading research. Such theories as we have 
iavohe intensely peisonal belief systems. Instead of looking for evidence 
to contradict (and thereby deny or exclude) a theory, we are prone to 



VVARDROP 

ERLC 



31 



25 



develop strong ego iiivolveineiu in a theory and to seek to assimilate 
evidence supporting it, sometimes even refusing to acknowledge openly 
contradictory evidence. Or, we have a pet theory and wish to demolish its 
competitors. We encounter considerable difficulty when we try to apply a 
strong-inference approach to a field such as reading. We seem to be faced 
with a choice lx?tween being softheaded, as has typically been the case, or 
disputatious. The hang-up is our tendency to idenufy theories with indi- 
viduals and— more crucially— to identify individuals with theories. Thus, 
to challenge a theory (or a hypothesis) is often tantamount to challenging 
the professional competence of a colleague. Piatt suggests a way out of the 
dilemma— a method attributed to a geologist named Chamberlin: 

"To avoid this grave danger, ihe method of multiple working 
hypotheses is urged. Ii differs from the simple working hypothesis in 
thai it distributes the [researchef s] effort and divides the affections. 
. . . Each hypothesis suggests its own criicrin, its own means of proof, 
its own method of developing the truth, and if a group of hypotheses 
encompass the subject on all sides, die total outcome of means and of 
methods is full and rich" [p. 3501. 

Piatt continues: 

. . . whenever each ni.in begins to have nuiliiple working hypotheses, it 
becomes purely a conflict beiueen ideas. It lx:conies niucli eusier then 
for eiich of us to aim every day ;ii conclusive disproofs— at strong 
inference— without cither reluctance or combaiivcness . . . [p. 350]. 

In considering the effecdveness of research, Piatt notes that: 

. . . the evident effectiveness of ihe systematic use of suong inference 
suddenly gives us ;i yardstick for thinking about the effectiveness of 
TresearchJ methods in general. Surveys, taxonomies . . . , systematic 
measurements and tables, thcoreiicnl computations — all ha\e their 
proper and honored place, provided they are parts of a chain of 
precise induction of how nature works ... (p. 351 j. 

We speak piou.sly of taking meiisurements and making small 
studies dial will **add another [)rick to the temple of .science." Mo.st 
such bricks ju,si lie around the brickyard (p. 351]. 

One last bit of ad\ice from Piatt pro\ides us with a guide to learning to 
appi) strong infcience to both planning our own research and evaluating 
anotlier's; 

h consists of asking in your own mind, on hearing an) .scientific 
explanation or theory , put forward, "Bui sir. what experiment could 
disprove your hypothesis?"; or, on hearing . . . [an] experiment 
de,scribed/'But sir. what h>|)()thesisdoes youi experiment dispro\er'" 
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I'liis goes Mraighl to the licai i of ilic iiiaticr. Ii forces everyone 
lo refcKus on the ceniral <|ucsiion of whether there is or is not a 
lesiable^cieniific step forward [p. 351]. 

Very little by way of "how to" advice is lx;ing offered here. Ques- 
tions like "How can I design a stud) which allows me to control for (x) 
while looking at the effect of (y) on (z)?" or *'How can I look at the 
relatiomhij) between (>) and (/.) independently of (x)?" are probably what 
most would-bc researchers would like to see treated. But we must invari- 
ably answer the '*how can I" questions by sa> ing, "It depends." There are 
some general strategies to apply to such questions, but the "Ix?st" answer is 
not always the same. It would be poindess to try to present in a brief 
discussion the wealth of detailed information needed to develop a re- 
search design. Once yuu have formulated youi (|uestion(s) and hypoth- 
eses very precisely {mk\ they will probably still be too \ ague), then it is time to 
talk about whether covariance or randomized blocks or matched groups 
or pretest-posttest oi some other approach fits best; about how to control 
foi order-of-presentation effects; about "how large a sample" should be 
utilized (which really means "How small a samplecan I get by with?"); and 
about all theudiei design issues that we need to consider when planning a 
lesearch study. The examples just used all relate to the internal validity of 
an experiment, the extent to which we ha\e eliminated potential sources 
of bfas. Another set ufciuestions has to do with external validity , the extent 
to which results from an experiment can be generalized. 

Research design to minimize errors of inference 

"How far can I generalize my results?" is one of the most frequent 
concerns of a researcher. The "proper" answer is disheartening: to only 
that population w hose members had a chance to be included in the sample 
from which you obtained your data, and only under the same conditions. 
In most reading research such an answer drastically limits external valid- 
ity (the tcim used by Campbell and Stanley in discussing issues related to 
generalizability). Most reading research— indeed, most social-science 
research— deals with what have been called "grab groups." That is, we 
study students in a single classi{K)m or school or district — oi maybe even 
two or three districts — or we expand our horizons to the level of a 
"statewide" study, but always our sampling procedures are guided more 
by consenience, accessibility, and economy than by principles of research 
methodology. Such constraints are, of course, unavoidable, so in the 
strictest interpretation of "generalization" we are effectively stymied. Jn 
the face of such a problem, we resort to what might be called— to contrast 
it w ith Piatt's "strong inference"— "weak generalization." The approach is 
best described in a 1968 article by Bracht and Glass, "The External 
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Validity of Experiments," Essentially, the argument is a nonscientific one: 
we are willing to accept the notion that our jxirticular sample is "like" some 
larger population with respect to those characteristics which we think 
might affect the experimental outcomes. Thus, even though practical 
considerations led us to use a "grab group," we are reasonably confident 
that we would obtain similar results with other groups from this larger 
population. It is on this basis that we cautiously generalize our results to a 
larger population than that fi om which we actually sampled. Needless to 
say, such generalizations cry out for replicating our research, each suc- 
cessful replication increasing our confidence in the validity of our 
generalizations. As an analogy in die physical sciences, consider the search 
for new chemical elements. A claim from one laboratory that a new 
element has been identified will not be accepted until another team of 
researchers at another location has used the same procedures and ob- 
tained the same result. For example, in 1926 element 61 in the periodic 
table had not been identified. That year, two University of Illinois 
chemists announced they had found it in ores containing elements GO.and 
62. The same year a pair of Italian chemists at the University of Florence 
thought diey had isolated the same element. But other chemists could not 
confirm the work of either group. It was not until 1945 that element 61, 
proinethium, was isolated among the products of the fission of uranium. 
Only then were other scientists able to replicate die work of three chemists 
at Oak Ridge National Laboratory. Su( h demands for replicability are 
virtually unknown in the behavioral sciences. 

There are some very real dangers in a weak generalization ap- 
proach. We invariably pay for our soft-headedness. In this case, the 
hang-up is the assumption that the target population (to which we wish to 
generalize) does not differ from the experimentally accessible population 
(from which we obtained our sample) with respect to "those characteristics 
which . , . might affect the experimental outcomes." In the first place, we 
^rc assuming— ohcn with good reason— that the target population is like 
die accessible population on the variables we consider relevant. More 
critically, we assume that we indeed know what the relevant \ariables are. 
Such an assumption is probably unwarranted in most reading research. 

Research design to minimize experimental error 

In conducting any research, we are always seeking precision. In 
odier words, we always want to obtain the most accurate estimates we 
possibly can of the parameters of interest. It is about this function of 
research design that we are most often concerned since most of us are 
prone to diink about research design primarily as it relates to statistical 
analysis. 
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Since it is impossible lo deal adequately u ith this aspect of design in 
a reasonable amount of space, this papei addresses Just a few general 
guidelines which help to minimize the ha.ssles involved in doing "good" 
research. 

Guideline 1: Keep it simple 

Just because elaborate designs involving many factors— Latin 
squares, split-plots, and incomplete blocks—have worked well in some 
fields (notably, agriculture) does nol mean that such designs are most 
appropriate for everyone. A common fault of doctoral students planning 
a dissertation (and some of us never outgrow it) is to equate good research 
with complex designs and elaborate statistical analysis. 

Guideline 2: Let the question determine the method 

This follows directly from the last comment. It may be because of 
the way staiistics courses are taught, but doctoral students seem to choose 
an analysis technique first and then seek to formulate a question to fit die 
procedure. 

Guideline 3: Avoid raw empiricism 

Raw empiricism only cluuers up the brickyard. Another isolated bit 
of data about a trivial question wastes time and lalent. If it is true thai 
people read to obtain meaning and that the purpose of instruction in 
reading is to enable the learner to obtain meaning, research in reading 
should be oriented toward understanding how meaning is acquired from 
printed sy mlx)ls, aftei u hich u ecan deal u ith the instructional technology 
issues with something stronger than an ad hoc approach. The more 
common research style seems to be to accept some particular view of how 
meaning is acquired and to focus the research on questions about die 
effectiveness of various procedures for leaching reading in a manner 
consistent with that view . About such research, we should ask a form of 
one of Piatt's quesdons, **But sir, w hat [imporiani] hypothesis docs [could] 
your experimeni disprove?" 

Guideline 4: Avoid empty theorizing 

A theory without empirical support is of liule value. More to ihe 
point, Piatt notes that "A theory is not a theory unless it can be disproved. 
That is, unless ii can be falsified by some possible experimenial outcome," 
A theory which can account for every possible outcome is empty, worth- 
less, A theory which does not lead to predictions subject to empirical 
disproof is merely an idle exercise. Ask Piatt's other question: "But sir, 
what experiment could disprove your hypothesis?" If you can find a 
substantive answei to this question, you have the basis for your next 
important research study. 
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Guidelines 3 and 4 advise us to avoid both raw empiricism and 
empty theorizing. Taken together, they emphasize the necessity for a 
continuing interphiy between theory and data. Without such interplay, 
research becomes sterile and lifeless; with such interplay, it is a lively and 
exciting experience. 

Guideline 5: Maximize exteiMial validity 

Reading researchers should be even more conscious than other 
social-science researchers that the phenomena to be studied relate to the 
functioning of real people in reaMife settings. Although studies in rather 
artificial laboratory settings play a role in contributing to our understand- 
ing of the reading process, it is not until our tlieories and methods have 
been subjected to the test of general application in the classroom and 
elsewhere that we accept them. One of the external-validity problems not 
given sufficient attention is what Egon Brunswik called "ecological valid- 
ity." In brief, if we seek theories and procedures to use in natural settings, 
wc must ultimately turn to those natural seaings for validating our 
theories and procedures. 

Guideline 6: Maximize internal validity 

Campbell and Stanley use "internal validity" to describe that aspect, 
of experinuMUal design which leads to the elimination of alternative 
explanations for empirical phenomena. In other words, design experi- 
ments in such a way that the only apparent explanation for positive 
outcomes is that uhich was hypothesized initially. In maximizing internal 
validity, we are guarding against the possibility that influences such as 
maturation, selection, and regression toward the mean might account for 
our results. 

Guidelines 5 and 6 highlight the inescapable conflict Ixitween 
internal and external \alidity. Standard approaches to experimental de- 
sign almost demand laboratory -like settings. Controlling extraneous 
infhienccs, minimizing experimental erioi, luling out alternative 
hypotheses~-these end.s cannot be easily cichie\ed except in laboratory 
settings. Considerations of generalizability of findings from research in- 
volving human behavior demand ecological \alidity, field tests under 
natural, uncomrolled conditions. 

Just wlicre the balance should be struck between internal and 
external validity will vary from one study to the next. Sometimes the 
importance of laboratory control outw eighs concei ns about generalizabil- 
ity; at other times, generalizability is the o\erriding concern. Finding an 
optimal compromise between these conflicting demands is the essence of 
good research design. 
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Cofisideratkm of yionexperimenlal research 

To this point, this presentation has focused almost exclusively on 
experimentation, in which one controls one set of vari*iblcs and looks at 
the effects of another set of \ ariables on some observable outcome. There 
is another large class of studies which should be dealt with, studies in 
which we examine relationships among variables in the absence of direct 
experimental manipulation. 

Although we tend to equate experimentation with one set of 
statistical-analysis procedures (exemplified by analysis-of-variance 
techniques) and associational studies with another set of analytic 
techni(|ues (typically correlational), there is no necessary correspondence 
between statistical technicjue and type of study. Corielational techniques 
are sometimes used in experimental studies, and so-called inferential 
statistics are sometimes used to analyze .issociatioiial data. Indeed, we 
have come to reali/.e in recent years that both sets of analy tical tools are 
specializations of the general linear model, and the essential e(|uivalence 
of col I elational and inferential approaches (to use the traditional labels) is 
now generally accepted. 

Two examples of nonexperimental approaches to research illus- 
tiate the role and importance of research design in such studies. One 
actively debated issue among leading researchers continues to be the 
(|uestion of whether reading is a unitary skill or whether it involves the 
integration of a number of component skills. The common approach to 
this problem is to administer a set of measures of hypothesized compo- 
nent skills to a group and then submit the resulting scores to a factor 
analysis. Sounds stiaightforwaid enough, does it not? Unfortunately, 
manuscripts reporting such investigations almost never merit publication. 

Why not? Let me suggest a few criteria to be applied to any study of 
this sort. Almost eveiy manuscript fails to meet more than one of these 
criteria. 

First, the investigator's choice of tests to measure the hypothesized 
skills is critical. Often the available standardized tests are i ejected as 
measuiing inappiupiiate skills oi (piesumably) having a factorially com- 
plex stiucture which vvould nut facilitate the identification of such unitary 
skills as might exist. The investigator then consti ucts tests specifically for 
the study. When this happens, one must look for evidence of content and 
coustiuct validity (including evidence that the items within a subtest are 
moi e like one anothei than they ai e like items in othci subtests). Ev idence 
foi internal-consistency leliability isalso es:>ential. On the other hand, the 
lesearchei may choose to use subtests from available standardized test 
batteries. In this case, one looks foi a clear and comincing rationale for 
the use of the chosen tests, a lationale which demonstrates the possihility 
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thaiilic li) pollicsi/cd skills will in fact be mcabiiicd, SimpI) put, the point 
ib thai the rcbulls o( a f acloi anal)bis ai c ci ilicall) dependent on the ijiialil) 
of the inpnt data, 

A second criterion relates to the number of different tests used to 
lepresent each h> pothesi/.ed skill, h is necessar) to reali/x- that tests are 
grouped tt)gethei in factoi analjsis on the basis of theii statistical com- 
monalities. In othei woids, if one has two tests of \eibal aptitude and two 
otheis measuiing fuigei dexteiit), the \erbal tests will be giouped to- 
gether as one iactoi . while the pei ft)riiiance measuies will ft)rm a sectmd 
factoi . Now , if one has a test of reading comprehension, a test of \erbal 
aptitude, a test of English usage, and a mathematics leasoning test (which 
t)picallv has consitleiable \ ei bal content), a ppl)ing a factor anal) sis to the 
resulting scoies is likel) to )ield a single "gcneiaT* factor of verbal facilit). 
Although the examples used to eniphasi/,e the point are extreme, the 
conclusion is equall) tiue when designing sinulai studies to identif) the 
skills in\ol\e(l in leading, iiamel). the stud) must include at least two — 
and pieleialjl) moie — different measures of each h) pothesi/.ed factor. 

A thiid criterion requires that a sufficientl) large number of sub- 
jects be tested. To opevationali/.e ^sufficiently large/* let me state that 
there should \k at least thiee times as man) subjects tested as there arc 
tests gi\en. If fewei subjects are used, theie seem to be some constraints 
on the test inteicoi lelations that make an) factt)i -anal) tic results (|ucs- 
tionable. 

A fourth ciiterion also relates to subjects. One must look for some 
e\idence that the) are fiom a cleai I) defined populatit)ii in which there is 
leason to belie\e that theie is \aiiabilit) among peisons with lespect to the 
skills being measuretl. Aftei all, what we arc tr)ing to do is infei some- 
thing about the sti uctui e of skills underl) iiig the i eading pi ocess for »>ome 
population, and since factoi anal)sis is based on corielational data, we 
neetl \ai iabilit) among the subjects in oi tiei to obtain meaningful coi i ela- 
tions to work with. 

Let us considei anothei approach to nonex peri mental research, 
the seaich loi causal lelationships in the absence of experiinentall) ma- 
nipulated \aiiables. Sociologists and economists ha\e been particularl) 
inno\ali\e in exploiing this aiea. l.i particulai, the techniijue known as 
path anal) sis has had considei able utilil) in i(lentif)ing causal patti^ins 
among sets ul \ariables. In thisappioach, one h)pothesi/es a set of causal 
lelationships among \ariables and deri\es a series of piedittions about 
I elationships w hich should be obtained among the \ ai iables if the model is 
appi opi iate. Those pi edictions ai e then testetl b) examining the correla- 
tions, regressiim weights, partial coi i elations, and/or partial regression 
weights actuall) obtained. Propel use of such approaches (and path 
aual)sis isoul) one of the se\eial techniijues a\ailable) i equiies Chainbei- 
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lin s multiple woi king liy potheseb. A researcher does not consider just one 
possible model; he considers alternative models and seeks to determine 
which one gives the closest match to the data. If you are interested in 
pursuing this topic, you will want to look at Maurice Tatsuoka*s discussion 
in the livsi Rrviexo of Research in Education and Herbert Blaylock*s book, 
Causal Models in the Social Sciences. 

The two examplesjust given (a factor-analytic study and the search 
for causal models) <ire meant to illustrate the need for as much attention to 
research design in nonexperimental as in experimental research. The 
\alue of a strong-inference orientation and the importance of multiple 
working hypoiheses apply equally for both kinds of research. 

Final remarh 

As die preceding discussion was meant to suggest, some research 
sii aiegies offer more promise than others. Good research requires that we 
begin asking good questions. The need lo challenge theories and hypoth- 
eses and coniinually seek to dis\>vo\Q them is essential. Experimental 
research genei. ill) offers more piomise for improving our understanding 
of reading th.in does nonexperimental. Final!), whether one is doing an 
experiinenial suid) oi a nonexperimental one. the importance of atten- 
tion to research design should not be underemphasized. 

If leading research were guided by Platt*s questions C'What exper- 
iment could dispro\e )our hypothesis?" and **\Vhat h)pothesis does your 
experiment dispro'. e?"). if researchers w ere to adopt Chamberlin*s notion 
of multiple woiking hypodieses, if designs were developed to maximize 
internal and external \alidity, ad\ances in our understanding of the 
reading process would surely be astounding, 

RKFERKNCKS 

B1.AY1.0CK. llOBhR \ M. Carnal mwlfh in thf sortal sctnirrs. Chicago. Aldinc-Athcnon. 1971 
UR/\C.ni . OUKNN 11.. & CLASS. O. V, The external xaluliiy uf cxpcnincnts Ammcan Edtuational 

Hfimrch Journal 1968. 5, 437.t7-t. 
C.AMIMU.1.1.. DONALD I X: S I AM.h\ , JULIAN C. hxffertmftitnt ami iiuau-expfntnental iUsigiu for 

uiMtch. Chir.igo: RaiuLMcNally. 
c:0X, DAVID R. Planmuffof fxprntnrtitu New York: Wiley. 19j8» 
HiXlTJCM IN R.Sm»ig inference. Sn^-Nfr, 196 L/-/6,3-J7-3r)2. 

I A rStOKA» MAO RICK M. Miilri\ariatc anal>Ms m wlutalional research In Kred V Kerlingcr (K<l ). 
Rntnoofffsrarch in fdueatton. Vol. 1. Itasca, Illinois: Peacock. 1073. 



W'ARDROI* 



33 



Measurement issues in reading research 



HAROLD F. BLIGII 
liarcQurt Brace Jovanovich 



The nationwide focus on reading has given rise to heahhy ques- 
tioning of current insii uctional methods and materials and to a search foi 
guidelines to impi o\e leai ning stiategies. \Vh jn reading research studies 
arecarefull) formulated, they provide valuable information for the kinds 
of decisions that need to \^ made, I lowever, in designingand implement- 
ing a reading stud) , the researcher must be alert to measurement issues 
which unresolved can lead to inconclusi\e fuidings. The purpose of this 
paper is to call attention to the kinds of measurement problems diat ma) 
arise and to offer some practical suggestions for dealing with them. 

To do good research, we must fust reali/.e thata problem exists — a 
problem that is worth) of the time and effort to be spent in stud) ing it. At 
least, the problem should merit seeking out information that w ill contrib- 
ute to a bcttei undeistanding of its ramifications. To adequatel) identif) 
and describe a reading i esearch problem, \\e must know how to ask good 
questions. 

Among areas to investigate are those involving theoretical formu- 
lations, model development, insti uctional strategies, learnei characteris- 
tics, and instrument development and v..lidat'.on. Some questions to ask 
are. What new theories of learning are emerging? Can reading theor) be 
applied in the classiix)m? How do children learn to read? Do all children 
follow the same behavioral patterns in leai iiing to read? 

In auempting to answ er such questions through planned research, 
we usual!) become involved in collecting information, quantif)ing and 
an<il)/ing data, and diawing intet pi etations and making generalizations. 
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Relevant infoi iiiatioii ma) be obtained tbiuugb talking with reading 
experts and searching the liteiature, b) c«irr)ing out earefull) planned 
experimental studies, and b) admiuisteiing tests and making t)ther kinds 
of (ontrolled t)bser\ations, Dr, Wardrop's article develops six guidelines 
for conducting good reading lesearch. He emphasi/.es the asking of 
meaningful i|ue.stions and the de\elopment of appn)priate designs for 
seeking ansvxers to the t|uestit)ns. We will next take a U)ok at some of the 
basic measurement issues invol.cd in planning and implementing read- 
ing research studies. 

Careful designing of the stud) eliminates, or at least minimi/xs, 
most of the basic measuiement issues, especiall) those associated with the 
anal} sis of data. All too f reipientl) , however, reading researchers proceed 
with data collecting and then attempt ttj search out appropriate ways of 
i|uantif)ing and analyzing the results of theii efforts. Sets of test scores or 
othei measurements collected in a hit or miss fashion, oi used because 
the) aie leadil) available, do not lend themselves to meaningful anal) ses 
and interpretation. 

I lowevei , no matter how carefull) a stud) is designed, (|uite unex- 
pected problems ma) aiise. As good researcheis, we are challenged to 
anticipate these pioblems and to be prepared to meet them. What are 
some of the majoi si>urces that ma) contribute to research error? In 
discussing the reduction of error variance, McNemar (1962, p. 382) 
classifies errors under thiee headings, "measuring oi observational er- 
rors, eirors in mfei ring population parameters in field oi surve) studies, 
and eriois in experimental testing of h)potheses." The first of these 
ei I ors concerns the reliabilit) of the instiument used foi data collection; 
the sectmd involves selecting samples that aie unbiased and representa- 
tive of the defined population, and the third deals with problems dis- 
cussed b) Wardrop in the Spring 1971 issue of the Reading Rvseauh 
Qiiarierly. Among the man) pioblems which continue to vex reading 
researcheis, Wardiop (p, 332) has identified thiee major ones that fre- 
quenU) appear in leading leseaich lepoits. These aie "design/anal)sis 
mismatch, inadeijuate specification of tieatinent, and contiol gioiips tliat 
do not *contror." 

Othei problems related to these basic categories are practical as 
well as statistical in nature, rietjuentl), the statistical pioblems are more 
I ead il) lesolved than the piactical ones. Fai i and Tuinman include some 
of the practical issues in theii article "The Dependent Variable: Mea- 
surement Issues in Reading Research,*' which appealed in the Spring 
1972 issue of the Reading Reseanh Qiuir/erly. These authors (p. 415) 
grouped measuiement problem.'- into four major categories, "the selec- 
tion and V alidation of appi opi iat^ measures, the i eliabilit) of assessments, 
the appropriate scores to use, and the desciiption of tests and testing 
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condiiionb." Thib list can Ik* exicnclecl to include problems invoked in 
identifying the popuhuiun and selecting the sample, in implementing the 
stud) , in assessing changes in reading beha\ ior, and in designing studies 
utilizing complex statistical techniques such as factor analysis and matrix 
sampling. 

In their very thorough treatise ''Experimental and Quasi- 
Experimental Designs for Research on Teaching," Campbell and Stanley 
(1963) identify twelve factors which may jeopardize the \alidity of various 
experimental designs. The) disunguish between internal validity dudexter- 
nal validity. In doing a controlled study, we may want to know if the 
experimental treatments result in a difference in performance of the 
subjects within the experimental setting. We would be concerned with 
internal validity and the factors which ma) operate to lessen the internal 
\alidity of the expeiiment. In planning the study and in drawing conclu- 
sions, we would want to know to what populations, what other settings, 
and which tieatment \ariable and measurement variables the observed 
effects can be geneialized. Here we are interested in generali/.ability . or in 
CamplK^II and Stanley s terms, external validity. Those authors go on to 
identify eight diffei ent classes of'extraneous \ ariables that are rele\ ant to 
internal \alidity and four relevant to external validity . These, in turn, are 
illustrated through presentation of sixteen experimental designs. 

Vxom this variety of factors, I have selected for consideration 
several which are relevant to m.iny kinds of reading research, and I have 
grouped them as follows: 

1. Some sampling problems 

2. Selecting the data-gathering instrument 

3. Problems in the field 

4. Selecting the appropriate score mode 

5. Measuring change 

Some sampling problems 

The sample is an integral component of the research design and 
has dii ect beai ing huxh on the choice ofstatistical techni(|ues to be applied 
in ti eating the data and {;n the generalizations that can be drawn from the 
results. Guidelines for sampling are provided in current statistics 
textbooks and journal articles. According to McNemar (1962, p. 382), 
"The aim is to secure a sample w hich is unbiased, that is, representativ e of 
a defined population, with chance sampling errors as small as possible." 
The problems of minimizing .sampling errois are related to the t)pe of 
study being planned. The errors which may arise in large scale sampling 
«is in survc) and noiming studies may be {|uite different fiom the sam- 
pling eiiois that may aiise when gioups aie selected for experimental 
purposes. 
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One of the fii st .steps in sampling is to define the population to be 
sampled. A rebcarchei investigating bome aspect of eai l> school reading 
might define the population as children entering the first grade during 
the fall of the school year. Subpopulations w ithin this larger one could be 
identific I as those children entering public schools and those entering 
private schools, or as those having had earlier school experiences and 
those with no school experiences. In selecting a sample that is to be 
representative of the poj elation and in identifying control groups that 
will operate effectively, consideration must be given to very practical 
factors such as the accessibility of subjects, the kinds of experimental 
conditions to be imposed, and the amount of time, effort, and money 
required to collect the desired information. If repeated measurements 
are to be obtained, mobility of the population fi om w hich the sample is to 
be drawn is an important factoi. Attrition is particularly crucial'when 
small samples are involved. 

Failure lo recognize the need for careful identification of die 
sample before starting to collect data is demonstrated in letters like the 
following: "Deal Sir; I am undertaking a research study in reading, A 
nearby school administered one of your reading tests in the fall. Could 
you please tell me if there is a second form available, so I can retest these 
pupils in the spring?" In this situation, economy of using an available 
sample seems to be of prime consideration. U|>on careful investigation, it 
may turn out that this intact group is most inappropriate for the purposes 
of the study. 

The size of the sample is also an important consideration and is 
directly related to the ty pe of research that is being conducted. To quote 
McNemar (1962, p. 84), "One of the aims of scientific method is to attain 
as great vision in results ^is is practical.** In experimental settings, as 
well as HI survey and norming studies, greater precision is usually ob- 
tained by increasing the size of the sample. However, it is how well the 
sample represents the population rathei than size, per se, that leads to 
meaningful generalizations. Through careful planning, it may be possible 
to attain an acceptable degiee of piecisi(;n without unduly increasing the 
size of the sample. 

The sampling unit is related to the size of the .sample and to the 
statistical techni(|ue to be applied in analyzing the data. Is the sampling 
unit to Ix: individuals draun randomly from a given population, or intact 
gioups such as classiooms? In applying analysis of variance, for example, 
data collected from clas.ses selected randomly should not be treated as 
tlioJJgh they were deiived fiom a random sample of individuals because 
classes not individuals are the sampling units. 

In defining the sample, all variables which may influence the re- 
sults of an expeiiment should be isolated and controlled. Consideration 
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should be given to possible effects of age, sex, grade placement, mental 
ability, type of school system, pre\ ions learning experiences, and ethnic or 
regional differences and related socioeconomic factors. For example, 
previous learning experiences ma> ha\e important bearing on the effects 
of treatments to be applied in a learning experiment at the second grade 
level. In population of grade two pupils, all pupils may not have had 
similar reading experiences during the first grade. The group might be 
quite heterogeneous in the sense that some had learned to read in a 
program using the linguistic approach, others in a phonics program, and 
the rest in a basal reader program stressing the whole word appnjjach. 

The final research report should fully describe the samples and the 
population from which they were drawn. The characterisdcs of the sam- 
ple have direct bearing on the kinds of generalizations that can be drawn 
from the results and are necessary information if the study is to be 
replicated. 

Selecting the data gathering instrument 

If a reading study is to provide meaningful information, the in- 
struments to be used in collecting the information must be carefully 
selected. The three basic characteristics to consider in choosing an in- 
strument are validity, reliabiiity, and nsabilUy. The first two of these are 
treated in most basic measurement texts and in the Farr and Tuinman 
article referred to earlier. The third characteristic, usability, encompasses 
problems encountered in administering the test. Such problems must be 
taken into consideration at the time the data-collecting device is being 
planned. If implementation of the study is to invoKe teachers or others in 
the collection of data, then the directions, timing, and other administra- 
tive procedures must be controlled so that errors resulting from the test 
administration itself are kept to a minimum. 

There are several sources for locating tests that have the desired 
characteristics. Ideally, an instrument designed specifically to fit the aims 
of the study will yield the most \alid information. Howe\er, development 
of an instrument involves extensive pilot work and field testing to ensure 
that it will function as desired. It generally takes from three to five years to 
bring a major reading test battery from the initial planning stage to the 
point where all the components are available for use. Usually, the time 
needed to develop a data-collecting instrument foi a reading research 
study is considerably less than this. However, anyone contemplating the 
development of a series of reading tests for a factor analytic study may 
find that solving the problems of item refinement, subtest reliabilities and 
subtest intercorrelations are the major tasks in the project. An interesting 
discussion of some of the issues involved in factor analytic studies of 
reading, can be found in an article by Thorndike entitled ^'Reading as 
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Reasoning" which appeared in the 1974 issue of the Reading Research 
Qiiarterly. 

A search of the literature may reveal the existence of reading 
instruments that have been refined and field tested in other research 
studies. Another convenientsource is the commercially published test. In 
examining read>-made instruments, the first consideration is that of 
validity . Will the instrument pro\ ide relevant information about the read- 
ing skill, instructional technique, or other variable that is being investi- 
gated? 

The title of a test or a brief description will not convey all the 
necessary information. A test of listening comprehension, for example, 
may have a response mode that is heavily loaded with reading, or the item 
type may be lapping skills other than those you plan to study. The test 
manual usually describes the rationale of the test and provides informa- 
tion on the validity and reliability of the test for its recommended uses. In 
addition loMudying the rationale and the procedures used in developing 
the test, the potenual user should examine carefully die item ty pes to see if 
they elicit the kinds of behavior the test purports to measure. By actually 
taking the lest in accord with prescribed directions, the reviewer may 
discover thai the methods used in presenting the stimuli do not lap die 
desired behavior or that the test is inappropriate for die maturity level of 
die children in the sample. In his text, The Specification and Measurement of 
Learning Outcomes, Payne (1 968, p. 52) presents a list of critical questions to 
be used in reviewing test items. His book, as well as others in die mea- 
surement area, provides practical information on designing a test. 

Articles w hich report on the use of the instrument in other studies 
may provide information about the reliability of the test with selected 
samples and may suggest ways in w hich the instrument can be improv ed if 
it is to be used in subseijuent studies. Bui os Mental Measurements Yearbooks 
are valuable sources of information about published tests. In the Year- 
books, brief descriptions of a test usually accompany reviews written by 
measurement and content specialists. These reviews, as well as those 
which have appeared in professional joui nals and reading texts, will help 
you to decide on the appi opriateness of the test for your study . Also, the 
test author oi publisher may have collected v.didity and reliability infor- 
mation subsequent to the publicauon of a test. When available, reports 
from these sources should gi\e you a better understanding of the test. 

Often, one or more subtests or a set of items from a research or 
published test may provide the kinds of iiifoi mation that you are seeking. 
A test may be lengthened to increase its reliability, or the directions 
altered to fit the conditions of youi experiment. If the test materials are 
copyrighted, pei misMon niuM be obtained before they can be adapted. To 
avoid delays and possible misundei standings, the request to adapt a test 
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slioiikl be |)icsciac(l to ihc.uithoi oi publisher dining the planning of the 
prujcci and Ix^foie final dcci^sioni aie made. All l(X) often, an urgent 
request conies in for permission to culapt copyrighted material foi use 
almost immediately, 

Reliabilit) of the measurements is the concern of all researchers. 
McNemai (1962, p. 382) states that "errors of measurement can be re- 
duced by developing moie leliable tests oi (when feasible) b) averaging 
repeated measurements." Since the leliabilii) of a test for a given situation 
is related to the variabilit) of the scores obtained, a test ma) not beecjually 
reliable in all situations. In de\ eloping or adapting an instrument, the 
researcher faces the problem of estimating the reliability of the instru- 
ment before administei ing it in the exjjei imental situation. If the samples 
reported inestimatingtheieliabilit) ofa published testdiffer significantly 
from those in )oui study, then the appropriateness of the repoited 
reliabilities ma) becjuestionable. In either case, reliability valuesshould be 
com|}uted and included in the final research report. 

'^roblem in the field 

Once the reading experiment is designed, the sample identified, 
and the ajjpi o\n iate data gat hci ing instrument is at hand, all that remains 
is for the leseaich to be implemented, It is in this phase that many of the 
more jjractical measurement-i elated problems arise, especially if the 
study is to be conducted in classrooms or other school situations. 

Careful arrangements must be made to insure that the subjects 
selected foi the sample will be available thioughout the entiie experimen- 
tal period; that appro|}riate facilities are available; and that teachers, 
parents, and, in as fai as possible, the children themselves undei stand the 
purjx)se of the study and the extent of their involvement. 

There may be times wiUiin the school yeai oi even within the school 
day that are inappiopriate foi the experiment you are planning. For 
example, the period between Thanksgi\ing and Christmas is usually a 
busy time for most teachers. 

Doesyoui project (all foi special training, a series of conferences, 
oi othei commitments f lom meinbeisof the schoolstaff ? If so, the nature 
and extent of these involvements should be made clear and agreements 
reached before the jjroject is started. 

Kven though thorough planning has taken place, administrative 
pioceduics h.ivc becMoutini/ed, and time schedules have been caiefully 
adhered to, unantici|)ated events will arise which may have an effect on 
the sample as originally defined , on the piocedui es for implementing the 
study, or on the interpretations and generalizations which may ensue 
from the results. Ociuii ences of this kind should be recoided and atten- 
tion given to the effect they may |}ossibly have. 
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Selecting the appropriale score mode 

The appropriate scoic to be Ubed in tiuanlifjing and anal)/.ing the 
information gathered in a reseaich btud^ dependb on a \ariet) of factors. 
The discuhsioii hei e is limited to the kinds of scores usuall) associated w ith 
standardized tests. Decisions as to the scoie mode to be used should be 
leached before the nistiumeiit isadnniiLsteied and the scoiesaie generated. 

If descripti\e or control data are to be transcribed from school 
records oi other sources, the type of score that is available and the 
completeness of the recoids should be in\estigated before the final deci- 
sion is made to use these data, It may happen that onl) deri\ed scores are 
a\aiiable when )ou intended to use ra\\ scoies, A plainti\e lettei like this 
one maj re\eal a pioblem which should have been anticipated: **Dear Sir: 
In carrying out my reading reseaich study, I planned to use IQ as one of 
the control \ ariables, To cut dow n on testing time and costs, I am using iqs 
which had been entered on the pupil lecords during the past two years. 
I'nfoitunately, 1 now find that about thiee-fouiths of the pupils took one 
intelligence test, the lest an intelligence test published by another com- 
pany , Do y ou ha\ e con\ ersion tables that 1 can use to equate these tw o .sets 
of ic>s?" Csually, there is little that can be done to help in situations like 
this. 

In geneial, raw scores *ne moie appiopriate statistically than de- 
ii\ed scores, Howevei, if alternate fbims oi sequential le\els of a test are 
administered, then scaled oi standard scores, when available, would be 
appi opriate. Raw scoi es and standard scoi es, how ever, may be difficidt to 
interpiet m a cuiiicular oi instructional sense. Fan and Fuinman (1972, 
p. 120) suggest thajt for piactical pui poses, the results be analyzed first in 
terms of law scoie dif feieiicesaiid then tianslated intogiade equivalents, 
percentile iank.s» oi other derived score differences. For example, a 
diffeience in giade equivalents of 8 months, oi a stanine difference of 2 
may be inoie leadily undei stood and applied than a law score difference 
of 12 points, Howe\ei , in inteipieting lesults in teims of derived scores, 
theieseaichci should be awaicofundei lying assumptions and any limita- 
tions of the derived score modes. 

Measuring change 

Reading iese»ncheis are -usually more concerned with investiga- 
tions in an .expeiimental setting than with large-scale oi normative 
studies. I hen designs sometimes call foi the identification of control «md 
experimental gioups and may in\ohe the assessment of changes in be- 
havioi nndei specified conditions. With lecent emphasis on accountabil- 
ity III eduuuion, theie have been incieasing attempts within schools and 
by external evaluatois to administci pictests and posttests and to assess 
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and nccoiini foi the aiiiouni of chdiige that has taken place during a 
prescribed period ofinsiriiciion. 

In a 1 est Service Notebook entitled Accouniabiliiy in Education and 
Associated Measurement Problems, Wrightstone, Hogan, and Abbott discuss 
measurement problems associated with accountability in education. 
These authors first define accountability and then lead into a discussion of 
two kinds of problems—those involved in determining the status of a 
pupil or group of pupils at one time during the year and those related to 
changes in performance over an insti uctional period. Academic growth is 
defined operationally as "the numerical difference of two sets of scores" 
(p. 5), Since this article identifies special problems associated with changes 
in acjideniic performance and offers possible approaches to their solu- 
tion, the points discussed are pertinent to anyone contemplating a gains 
study. 

What are some of the issues that should concern a researcher 
interested in assessing changes in reading beha\ior? In his text, Educa- 
tional Mea,sumnent$ and Their Interpretations, Davis (1964) provides some 
basic guidelines when change is to be measured b> subtracting pretest 
scores from posttest scores. He states, "This method of measuring change 
is appropriate if the test forms are highly reliable, if the pupil or pupils 
were not selected for training on the basis of initial score, and if the 
estimate of change for the pupil or groups of pupils is not to be compared 
with changes in the scores of pupils or gi oups of pupils whose initial scores 
were markedly different/' He continues, 'The best measure of change in 
an individual is an estimate ol'ihc true change" (p. 251). 

Among the problems associated with measurement change is that 
of determining the amount of time to intervene between testings. There 
must be sufficient time foi learning to take place if the differences are to 
represent real gains. If the time interval is relatively short, or if there are 
other uncontrollable factors w hich may affect the test scores at the end of 
the learning experiment, then alternate forms of the test should be 
administered at the beginning and the end of the experiment. If this is the 
plan, it must be determined in a(hance that altei nate forms are available 
and Ihai they are comparable in design and content and yield ecpiivalent 
scores. This issue is of real concei n to researchers w ho develop their own 
instruments or adapt standardized tests in such a way that pre-established 
interform relationships no longer hold. 

During the preschool and primary years, reading skills may be 
developing so rapidly that the level of the test administered at the begin- 
ning of the experiment may not be appropriate as the final test. If a 
relatively easy test is given at the beginning of the assessment, there may 
not be sufficient ceiling when it is used as a posttest. In developing or 
selectinga multilevel instrument as a solution to this issue, the problem of 
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developing scoics e{iiii\alcnl fiom level to le\el has to be recognized and 
resolved. 

Implicit in the Davis guidelines and the concerns e> pressed by 
Wrightslone and his coaiilhois are the issues of regression toward the 
mean and reliability of difference scores. When the same test scores arc 
used to place pupils in the lesearcli piogiain and again as pretest scores to 
be compared later w ith posttest scores, the problem of regression occurs. 
The differences in scores leflect not only the real gains but also the effects 
of any regression that may have taken place. 

Fortunately, problems of regression can be avoided through care- 
ful planning. One appioach is to identify pupils for the research program 
by some means other than the test used in gathering the data. A different 
test or a different set of ciiteria can Ix; used to select the students. Also, 
when careful attention is given to matching design and analysis 
techni(|ues, the pioblems of measuring change are reduced. The effects 
of regiession can then be controlled through the use of appropriate 
statistical techni(|ues. 

Problems of reliability of gain scores have been vexing reading 
lesearcheis loi years. In a guest editorial for the Spring 1971 issue of the 
Reading Re^eanh QitaTterly, Stake and Wardrop vividly illustrate that the 
unreliability of gain scoies foi individuals can give the appeaiance of 
learning vx hen no learning has taken place. Many statistical texts cov er this 
topic in detail. Both Davis and Wrightstone present examples which help 
to clarify the concepts of regression and reliability of difference scores. 

There is no simple approach which will resolve all the measure- 
ment problems which may arise. This discussion has recognized the 
importance of formulating the problem and asking appropriate questions 
.and the importance of a design that will provide reliable and valid infor- 
mation about the problem. In addition, measurement issues typifying 
those that many i eading researchers will face have been identified. How- 
ever, there are others which may be more significant to a specific research 
project. Good researches must anticipate practical as well as statistical 
problems that may arise and be prepared to cope with them. 
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Conducting longitudinal research in reading 



KATHRYN A. BLAKE 
JERRY C.ALLEN 

University ofGeorgid 



Description of longitudinal research 

Longiuidinal research is one method for studying how and why 
pupil iK'havior changes over a relatively longtime period. In longitudinal 
research, we have two purposes. The first purpose is lo describe how pupil 
behavior changes as time passes or as pupils participate in particular 
treatments as time passes. That is, we analyze and portray the nature of 
the curve, showing rate of behavioral change as a function of dme or ume 
and treatments in combination. The second purpose is to ascribe those 
behavioral changes to parUcular independent events which occurred 
during the lime span studied. That is,, we interpret the behavior we 
described. 

Longitudinal studies are often called growth studies or develop- 
mental studies. Grow th studies are \ er> important in gaining information 
to increase the effectiveness of education. As van Dalen (1973, p. 241) 
pointed out: 

lo teach cncctivcK . one must have knowledge of the nature and rate 
of changes that take place in lunn.ni organisms. One must know what 
mierrelaicd factors affect grouth at \arious stages of (le\'elopnicnt; 
when various aspects of giowih are first obscrvaljle, spurt forward, 
leinain niilier .stationary, reach optimal (levelopmenr., and decline; 
and how the duraiion, intensity, and timing of an experience in the 
dcx'clopmenial period affect growth. 

Longitudinal research is only one method for studying how be- 
havior changes over a longer period of time. Other methods are the 
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cross-scctioiial iiictliod, ilicscmilongiuiclinal method, and die retrospec- 
tive method. Let's look at longitudinal research in relauon to these odier 
three methods. 

The longitudinal method invoh'es making a number of repeated 
observations on ihe same subjects as they progress through a number of 
time periods. For example, to study pupil growth in interpreting figura- 
tive language, pupil response to idioms, metaphors, similes, and other 
devices could be sampled when the pupils are at Grade 1, Grade 2, Grade 
3, and so on. Then, this group's average scores for each successive year 
could be joined to make a growth curve. 

The cross-sectional method involves making one observation oudiffer- 
ent subjects at a number of levels. For example, to study pupil growdi in 
reading compreliension skills, measurements could be made at one given 
time for a group of pupils at Grade 1 , another group at Grade 2, another 
at Grade 3, and so on. Then, the several groups' average scores could be 
joined to make a growth curve. 

The semi-longitudinal method is a combination of die longitudinal 
method and the cross-scxlional method. It involves measuring the same 
subjects at a given level at least twice o\ er a period of time (and sometimes 
more) and measuring r/Z/jTrnvi/ subjects at different levels. For example, to 
suidy pupil growth in sight \ ocabular) , measurement could be made in 
the fall and the spring of a gi\'eu year for a group of first graders, for a 
group of second graders, a group of third graders, and so on. Then, each 
group's curve for a year could be joined to other groups' curves to make a 
long-term growth curve. 

The retrospective approach involves using reports of past behavior 
rather than observations of present and future behavior. People, such as 
parents, who have known the pupils describe how their behavior was at 
various times. For example, to sliou pupils' early progress in reading, the 
parents could recount the ages uhen their children first became attentive 
to words; when they first started to read road signs, labels, and similar 
materials; their initial responses to instruction in school; and so on. 

All of these approaches are useful in dealing with questions about 
longer-term Ix^havior changes. Each has its \cilues and its liniiialioiis. The 
topic for this paper is longitudinal research and that is what we will 
consider from here on. We will look at illustrations, concepts, interpreta- 
tions, and research questions and designs. 

Illustrations of longitudinal research 

Among die classic longitudinal studies are Bayley's research on 
intellecuial growth from infancy through old age and iheTerman group's 
resc^cMch on highly intelligent children's growth in various areas from 
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childhood through old age. Instances of longitudinal research in reading 
include work b> Bond and Dykstra (1967), Bruininks and Lucker (1970), 
Mazurkiewicz (1971), and Satz (1974). More broadly, the historical 
perspective is shown in Stone and Onques survey (1959) of time-series 
research for the past one hundred > ears. A curient example in education 
is the work of the American Institutes for Research (A.I.R.) with Project 
Talent subjects who are now in their 30s. In 1960, the A.I.R. group started 
with about 400,000 randomly selected pupils. The plan is to observe 
students at various periods after their high school gr^aduation to see how 
much the education and counselling they received helped them in later 
life. The National Institute of Education recendy granted funds for 
another stage in the study. 

To get a clearer picture, let's look in more detail at Bayley's lon- 
gitudinal w ork on the grow th of intellectual abilities from infancy through 
the life span. The research has yielded some information which has 
important implications for reading instruction. For example, her infor- 
mation about adult intelligence has implications for reading improve- 
ment in post-high school students and othe: adults. In the past, one 
notion has been that intelligence is a unitary, or general, ability in which 
growth levels off in late adolescence. If this were so, then we could 
h>|XJthesize diat growth in readingskilLs related to intelligence would also 
level off in late adolescence. However, Bayley (1966, 1968, 1970) has 
shown that intellectual growth continues into adulthood, that there are 
different intellectual abilities which show different patterns of growth, 
and that males and females show different rates and levels of intellectual 
growth. The growth curves in Figure 1 show these patterns of develop- 
ment in intellectual abilities during the age period between la 16 and CA 36. 

Bayley (1970) made these comments about her data. 

From ihese suidies again we have evidence for niuhiple menial 
abiliiies, which develop in differein wa)s. Some slum more conrinu- 
ons growdi dian others. Sonic are more consisieiu over rime dian 
oiheis. U appears that one general class of abilities, which may he 
leferred to as verbal facilit) and knou ledge, is not onl\ more stable 
uit hni indu idnais ihi oiighunt grou th bnt alsoconiiniics to increase in 
adnlts to !J0 years of age oi older. Other abilities ap|xrar to be more 
bonnd to stage of devclopmein. to l>e less stable over time, and to 
I each theii peak in the luenties. Such fluid abilities include reasoning 
processes, arithmetic and verbal reasoning, perhaps attention span or 
short-term memory and speed [p. 1 185]. 

To the extent that abilities sampled by the Wechsler Scales aix? related to 
reading skills, Bayley s information indicates that we can hypothesize: that 
older adolescents and youngei adults could continue to develop in 
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reading-- that is, to shou higher le\els in acquiring comprehension, in- 
terpretation, evahiation, and processing skills in reading and in learning 
to apply those skills in dealing with problems— and that males and females 
would 5how different patterns of learning such skills. 



Concepts pertaining to longitudinal research 
Questions in longitudinal research 

Longitudinal research can be used in answering several types of 
questions. In this paper, we will consider design, analysis, and interpreta- 
tion for two types: questions about hou a particular characteristic changes 
over a period of time as students grow older or participate in certain 
common experiences with time, like schooling; questions about how a 
characteristic changes as students participate over a period of time in 
particular treatments, that is, any instructional procedure or other ex- 
[XM-ience designed to cause changes in behavioi . These types of questions 
can be asked with a cross-section of students. Or they can be asked about 
particulargroups of students whoare selected to bedifferenton a particu- 
lar subject characteristic and homogeneous on other relevant characteri.s- 
tics, for example: males and females; pupils from English speaking 
backgrounds and pupils from non-English speaking backgrounds. 

Changes xoith lime 

Does pupil behavior change over a period of time? 

Do different groups of pupils show different patterns of behavior 

change over a [XM iod of time? 
Changes xoilh lime and Irealmenls 

Does a given treatmentaffect pupil beha\ior over a period of time? 
Do different groups of pupils respond differently to a given treat- 
ment over a period of time? 

These questions, of course, are stated in general form. In particular 
research projects, they would be more specific to reflect the situation. 

Designing longitudinal research 

There is a research design for each longitudinal research question. 
A research design is a plan for collecting data needed to answer a ques- 
tion. It shows the arrangement of the variables. 

The independenl xwiahle is the variable whose influence on a depen- 
dent variable is IxMug studied. In longitudinal research, time is the main 
independent variable. As Holtznian (1963) commented: 
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Any approach lo ihe siudy of change niusi involve two or more 
nieasurenienis oi the same variable ai different times in order lo 
pro\ ide a basis foi inferi ing l hat change has or has noi laken place, I n 
all such designs, time is the niasiev xariable ag-ainsl which ever) thing 
else is ordered [p. 199]. 
Other independent variables are subject \ariables and treatment vari- 
ables. Subject variables arc characteristics of the pupils. Stibject variables 
of interest in reading research include such characteristics as visual acuity, 
auditory acuity, intellectual ability, meiual maturity, language facility, 
cognitive style, motivational style, and so on through a wide range of 
variable.s. Treatment variables are methods, materials, and other proce- 
dures designed to influence the dependent variable. Treatment variables 
in reading research inckide such procedures as emphasis on decoding 
skills, alternation of type si/e and style, use of advance organizers, ma- 
nipulation of redundanc), timing of prereading experiences, variation of 
syntactic structiues, airangement of generalizations and specific in- 
stances, and so on through myriad pssibilities. 

The dependent variable is the variable which may be influenced by 
the independent \ariable. In reading research, these could be any aspect 
of reading behavior or any aspect of behavior related lo reading, for 
example, skill in \isual discrimination, knowledge of phonics rules, un- 
derstaniling of the meaning of unboimd morphemes, grasp of concepts, 
memory for sentences in various transformations, skill in finding ideas, 
facility in detecting relations, familiarity with literar) genre, and so on 
through a very large number of variables. 

Control variables are the variables which are not under study but 
which could enter in to affect the dependent variable and lead to errone- 
ous conclusions about the influence of the independent variable. These, 
of course, vary with the independent and dependent variables being 
studied. 

As with other research approaches, the chief criterion for an 
adetitiate design in longittidinal research is. The procedures for dealing 
with all of the variables should be stich that the data obtained will yield 
imequi^'ocal, valid answers to the research tiuestion about the indepen- 
dent and de[>endent \ariables being studied. In later sections, wc will 
consider threats to validity which can operate in several designs. If we plan 
adequate ways U) circumvent these threats to \alidit) and if we can dem- 
onstiate that we did circumvent them, then w e can make valid generaliza- 
tions following from the research question. 

Analysis of longitudiTial data 

Once we obtain our data, we are ready to analyze them. As noted, 
the first purpose of longitudinal research is lodeuribe change which takes 
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place in the cicpciiclciit \ai iablc as liinc passes or as pupils are exposed to 
particular treatments as time passes. We seek to make four kinds of infei- 
ences when ue describe change with time in longitudinal research. Here 
are the inferences accompanied by the e\idence stated in statistical and in 
non-statistical terms. For illustration, the material is keyed to Hgure 1 
showing some of Bayley's longitudinal research results. 

T 0 make inferences about ivhet/ier a change occurs. Does the slope of 
the curve differ from /xro: Is there a trend? Does the dependent variable 
change? Note that Bayley's cur\es are not flat. All have slopes which 
depart from zero. We can infer that intellectual abilities do change be- 
tween CA S 16 and 36. 

If a change occurs, to make inferences about the nature of the change. Is 
there a linear trend. Is there an e(|ual change in the dependent variable 
for each change in time? Is there a ([uadratic trend; Is there an unequal 
change in the dependent \ariable for each change in time? Is the rate of 
change posiii\ely accelerated, that is, increasing with time? Is the rate of 
change negatively acceleiated, that is, decreasing with time? Is the trend 
cubic, is it (|uartic, etc.. Is there more than one inflection point where rate 
of c hange varies as it happens w hen pupils grow , reach plateaus, and start 
growing again? Bayley's curves suggest that female.s generally show a 
linear trend in vocabulary growth between c>As 16 and 36. We can infer 
that they show e(|U<il increments in vocabulary between testing periods. 
The other trends are'non-linear. During the 20-yeai period, the groups 
show various patterns on increments, plateaus, and decrements on the 
different abilities. 

T 0 make inferemes about ivhethei subject gioups or treatment groups shoio 
similar trends. Do the slopes of the curves differ in displacement; Do the 
groups differ? Are the slopes parallel. Do the groups show the sam.e rates 
and patterns of change? Bayley's curves suggest that males and females 
show similar patterns of growth on the characteristic sampled by the 
Information Scale and different patterns in the other abilities. Note 
especially the two groups' relative positions and patterns on Arithmetic 
a vis the othei scales, especially Vocabulary, Similarities, and 
Comprehension. 

To make inferences ot projections about hoiv trends ivdl continue. Given 
a trend, what values can we expect ihiough given future time periods? In 
no case did Bayley^s groups reach the top of the scales. Consider what we 
might expect fiom the next 10- or 20-year periods in abililies like vocab- 
ulary in contrast to abilities like innnediate memory (Digit Span). 

We base these inferences about concrete behavior on data at two 
levels of abstraction. At the first level are scores on instruments used to 
sample the dependent vaiiable and, in some situations, the independent 
and control variables. Wc obtain these data by measurement procedures 
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dcMgnated |>s>dionielii( pioccdiii cb. At the second level of abstraction 
are statistical indices \\c obtain b> analyzing die scotes witli particular 
statistical procedures in four categories, procedures for describing indi- 
\iducds, correlational procedures including regression analysis, analysis 
of variance pioceduies including co-variance analysis, and multivariate 
procedines. 

Each psyclioinetric .ind statistical procedure has its requirements, 
values, and limitations. Problems in psychometric procedures and statisti- 
cal procedures compound uith problems in design procedures to create 
some majoi difficulties in getting unequivocal chough data to accurately 
descrilx: the change uhich lakes place in reality. In short, different com- 
bmations of piocedm es used to describe die same phenomenon can lead 
to different lesults. Therefore, procedures need to be weighed and cho- 
sen carefully to get the soundest results possible with the fewest restric- 
tions. Bereiter's comments (1963) illustrate some difficulties: 

rhe urilcr [Bcrcilcrl had ihc pii\ilcgc of working with the Vassar 
smdics of pcisonalily and auimdc (haiigcs (luring (he college years. 
Moslof lhci|uaiuiuii\ed.ila ucrc obtained from iwoadminisiraiions 
of the same lesl l)aucr>, once in the freshman year and once in die 
senior \ eai . ^jonie pi onounced genei al changes u ere observed, and a 
naunal question toask was whet hei these changcsoccuriedaian even 
rate ihroiighoiit the foin years or uhethei they were most!) accom- 
plished during the fnst yeais. Since yearly iciesting of the same 
pel sons could piobabi) have a sei tons (oniaininating effect, samples 
of one class weic retested at \.nious times — some at the end of their 
freshman >eai , sonic at the end of theii sophomore year, and so on. 
As often hap|x;ns, ho\\e\er. the subjects, though in\ited at landom, 
(lid not accept laudonily and it turned out that samples retested in 
diffeient yeais differed significantly in freshman scores. What to 
make of this? We (ould igiioi c the disci epancies bctu een initial scores 
andnierel) plot theietesl means foi the \aiious groups as a function 
of time of letestmgaiid ti > to fiia toiitinuouscur\eofcliangetoit.Or 
wetould.b) any nunilx:rorde\ices.attenipt toadjust the relest scores 
to (oinpensatc (oi differences in iiiiti.il scores. Depending on the 
detision, we tould gel the tmves to kK)k almost any way we like- 
linear, negatively attelerated, S-shaped, or what have you. 

In anothei pait of the Vassal studies we were concerned with 
(111 Terences among cuuiculai groups. Two (|uestions were of interest: 
,1) Do dilfeicut fiekls attiact different kinds of students? and b) Do 
students tlumge diff ei eiiil> dm ing college depending on their major 
field of study? The answei to the fuM (luestion was unequivocally 
"yes" and this made it difficult to aiiswei the second (juestion. Several 
related measures of "attitude giowth" were available. When raw 
change was used, it mrned t)Ui that the /mm groups weie initiall), the 
more ilie^ gained dm ing ( ollege. When the initial scoie on any one 
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mccisurc ivns partlaled out, it was found that tlic/i/g/ifr groups were on 
related measures, tfie nioic they gained. Tliis complete reversal of 
findings when the switch is made from raw to corrected change scores 
seems to be typical, and it is difficult not to be suspicious of it, to 
suspect that it represents a switch from one kind of error to anotlicr 
tp. 4]. 

The point here is that to perform or evaluate longitudinal research, you 
need to be familiar with such problems. You can gain such knowledge in 
psychoineirics or statistics courses and in books like Annasiasi (1968), 
Cronbach (1970), Edwards (1968), and VVeiner (1971). 

Interpretation of longitudinal data 

As noted, the second purpose of longitudinal research is io ascribe 
the changes we observe in the dependent variable to the independent 
variables, that is. to particular events which occur during the time span 
studied. Such interpretation can be rather tricky business. As Campbell 
(1963) pointed out: 

We have examined the methodology of asking; "Did a change take 
place heie?" "WIku attributes changed?"' "WIku things tend to change 
together?" "Wliat is the general, normal, change or growth pattern 
for this animal, this person, this gioup, this institution?" "How do 
change patterns vary as concomitants of social settings?" The prob- 
lems encoiuuered at this level are so formidable that it may seem 
gratuitous to ask a still more ambitious one. So frecjuently do data 
processing procedures mislead us in answering the (juestion "Did a 
change occur?" that we should perliaps hesitate beforejinnping to ask 
"What caused the cliange?" 

Yet, in the literatm e. the methodology which we are seeking to 
improve, this reticence is usually lacking. Few describe change with- 
oiu attempthig to explain it. Most undertake the research assuming in 
advance that the> know the causal agent, asking onl) the question of 
the degree of efficacy: be it of genetic patterns, of maturation, Vassar 
College, niiliiary experience, psychotiierapy, remedial reading, or 
what not fp. 212], 

Interpreting longitudinal data is the same as interpreting dam 
obtained with other research methods. VVc can accept hypotheses about 
the causal influence of our independent variables only when plausible 
rival hypotheses have been ruled out. That is, once wedesctibo the relation- 
ships w}iich exist and then seek to ascribe them to otir independent vari- 
ables, \ve^o through three steps. 

• Formulate hypotheses about all of the variables in the situation 
which could cau.se changes in the dependent variable. These 
hypotheses include li>pothese.s about the independent variables. 
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And these hypollicses iiiciiule rival liypotlieses aboui other, ex- 
traneous variables which could be influencing the dependent vari- 
able and thus be a threat to the validity of the study, that is, to the 
extent to which the data will yield unequivocal answers to the 
research <|uestion about the independent variables and dependent 
variable being studied. 

• Present evidence to discouiu the rival hypotheses. Given sufficient 
evidence, reject these rival hypotheses. 

• Accept the hypotheses about the independent variables. 

To be able to carry out these three steps, we need to plan ahead 
when we are setting up a longitudinal study. We need to anticipate what 
variables in addition to the independent variables could enter in to 
influence the dependent variable and thus become rival hypotheses or 
threats to the validity of the study. Then, we need to use appropriate 
procechires to deal with these potentially problem variables. Several au- 
thors (Campbell and Stanley, 1963, pp. 171-241; Bracht and Glass, 1968, 
pp. 437-474; van Dalen, 1974, pp. 259-361) have described variables to 
consider and procedures for dealing with rival hypotheses or threats 
validity. 1 he material below describes some \ariables or threats to internal 
validity which need to be planned for in the longitudinal research designs 
involving groups and treatments described l^elow. (To keep down the 
complexity and length of thib chapter, threats io external validity are not 
discussed herein. However, they merit consideration.) 

History, Sometimes, specific events take place between the times 
measurements are made, and these events operate in addition to the 
independent variable to influence the dependent variables. For example, 
suppose we were studying the effect of the Frostig program for training 
visual perception on die development of young rhildren's visual discrimi- 
nation performance. If the children were concurrently watching 
Street, we would not know whether to ascribe changes to the Frostig 
pro^rdm.Scsame St)e€t, or both. Dealing with history as a threat to internal 
validity involves two procedures: 

• When tieatineiit.s are being studied, use a control group who (I<)es not get 
the treatment but who would experience the specific events. Differences 
l)et\\een the ti eatnient and control groups could then be attributed to the 
treatment, other things being e<|nal. 

• In studies involving treatments and in those not invohing treatments, 
specify variables which could enter in to confound the results; explicitly 
survey whether the subjects experienced those variables; and, if so, limit 
generalizations appropriately. 

Psychological changes. Over time, subjects may change in ways 
which have nothing to do with the independent variable. That is, they may 
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lose intei est in thcsttid), tlicy in*i) .icqiiiiea particular enthusiasm foi the 
study, and so on. Foi example, suppose we used an audio\isuaI program 
for teaching \ocabular> that had a large no\elt) appeal for younger pupils 
but not for oldei ones. Changes in the dependent \ariable might decline 
o\er the time period because of differential interest, not because of 
decreasing effecti\encss of the program. Here are three procedures 
appropriate to deal wiUi psychological changes as a threat to internal 
validity: 

• hi treatment studies, uj>e a control gn)Up which does not take part in the 
irealnient hut does partake in eiinall) uiteresting parallel acti\ities. 

• III both tieatiiieiii and in iioiitieatnieiit .studies, identify attitudes and 
iiitciest.s which iiiiglit iiiflnenie the dependent \aiiable. survey them 
se\eial limes during tlie study, and, if they operate, restrict generali/a- 
lions appropriately. 

• In all studies, choose tieatinent iiietliods which would not be subject to 
changes in interest, etc. If possible, spend a warm-up period giving 
subjects expeiiciice with tliesiiuation so that the lunelty would dissipate 
before the study starts. 

Ri'sffomcs to tests. term test may be used broadly to refer to .any 

procedure foi sampling the dependent \ariable, foi example, papei and 
pencil instruments, latings, inter\ieus, mechanical devices, etc. Some- 
times, testing influences subjects in a way that affects their later responses 
to the tests: they may learn \\hat is expected, they may be traumatized, 
they may ha\e theit inteiest aioused, and so on. Foi exatnple, suppose a 
reading test gi\eii first to beginning first gtadets in\oh ed pupils' making, 
for them, a fairly complex i espouse like completion oi filling in the 
blanks. The pupils could scoie spuriously low on the first test and then 
higher stibsequendy when they learn how to i espond. Such impro\ement 
could not be asciilx-d correctly to the independent \ariable. Procedures 
for dealing w ith testing as a threat to validity include: 

• Use the same test throughout the study if possible. 

• ri.uii pupils in the nicchaiiics of iespoiiding to the tests before the tests 
aie administered in the research situation. 

• Use control subjects who recei\e the tests but not the treatment. 

Changes in tests.^ 0\ er time, tests may undergo \ arious changes like 
changes in calibiation, obsei\ers, alternate forms, and re\isions. Such 
changes may affect die obtained measures of the dependent \ariable. For 
example, children might be gi\en one form of a reading test when they 
are at the primary reading le\els and another form when they are at the 
intermediate reading le\els. If the two foims diffeied in relative 
difficulty, differences in the dependent variable could not be ascribed 
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eiitiif ly to the iiidcix^nileiit \aiiablc. Scvcial pioccduics for dealing with 
changes in tests as a threat are: 

• Calibrate mechanical devices to a nonn bcfoic each testing period. 

• Use standaul pioceduiei) fui i ate is and observeis and have training 
bessioni) at the uulscl of each testnig petiod. Assign Mibjccls raiuluinl) to 
raleis and observers oi vice versa. Be sure all suiyecis are e.xposed 'to * 
diffeieni reseauhers over time oi conversely that no researcher deals 
e.xclusively with a particular group ofsubjecis. 

• Use only jlleniale test forms v\hich arc demonstrated to he e(|uaie(l. 

• Use tests uhei e subse<|ueni foi ms are carefull) calibialed to shade into 
one another and not vary in relative difficulty. 

• U.se coniiol 5ul)jecis who receive the tests but not the ireainients. 

Statistical regression. When groups have been selected on the basis 
of scores which are above the mean or average, the scores they obtain 
when they aie re tested will move lower toward the mean. Groups chosen 
for the scores below the mean will show the reverse pattern. These 
changes are manifestations of statistical regression. The) might seem to 
be the effect of the independent vaiiable, however, the) occur because of 
psychometric and statistical phenomena. For example, suppose a group 
of poor readers were chosen for a lemedi.d treatment, with poor re«iders 
being defined as pupils two grade-levels below the average in reading 
achievement foi their age group. On retesting, their mean scores could 
move toward die average i egardless of the influence of the independent 
variable — the remedial reading piogram. Dealing with statistical regres- 
sion as a threat includes procedures like the following: 

• In treatment studies, use a control group. 

« In stu(lte^ not involving treatments. attiit}Ute the changes toward the 
population mean to the possible influence of statistical i egression in 
addition to the influence of the independent varial)le. 

Differential selection of groufts. Groups may be used in two ways: in 
studies of independent subject variables and in studies of treatments. 
Gioups used in these wa)s must show ceitain similarities and differences 
on variables related to the dependent variable: 

• When groups are used to study the effect of an independent subject 
v.iiia!)k' on a dependent vaiiat)le» these gioups should differ on the 
independent vai table and be eipiivalent on all contiul variai)les» that is, 
on all iithei subject charactei istics i elated to the dependent variable and 
all e^perience^ v\hich could influence the dependent variable. Other- 
wise, given diffeienceson these other variables, these differences might 
eiitei in to mflueiicc the dependent variable and confuse the results. For 
example, su|)f)usc v\c v\cic tompai ing the reading giowth of pupils high 
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in need achievement with the reading growth of pupils low in need 
achievement. Dillerences in reading growtli could not be attributed to 
difTerences in need acliieseinent level if tlie pupils also differed on 
related variables like initial status in reading— as they very well might. 
Dealing with this threat involves steps such as the following: 

— Identify possible control variables related to the dependent variable. 

— Get evidence aboiu these variables. 

— Demonstrate statisticall) and/oi judgmerually that the groups do not 
differ on these control variables. 

• When groups are used in a study of a treatment variable, they are lalx-led 
treatment (or e.xperiniental) and control groups. The treatment group 
gets the treatment; the control group does not gel the treatment, but it 
docs get .1 parallel, coiurol activ ity . The groups must be e(|uivaleni on all 
subject variables related to the dependent variable and on all experiences 
except the treatment or the control activity. Otherwise, differences Ix-- 
tween the tieatment and control groups on the dependent v ariabic could 
not be attribiaed to the treatment. For example, suppose we were ex- 
amining the effect of prereading activities on subsequent reading 
achievement in thep;imary grades. If the treatment and control subjects 
dif (ei ed on some related \ ariable, like intelligence lev el, the effects o( the 
treatment could not be evaluated. Dealing with this threat involves pro- 
cedures such as the following: 

— (let a pool of subjects and assign them to the treatment and control 
groups by appropriate procediues (e.g., van Dalen, 1973, pp. 274- 
279). 

— Use statistical and judgmental evidence to demonstrate thai the 
groups arc c^juivaleru on all vaiiables which might affect the dejxrn- 
dent va liable. 

Differential subject attrition. Attrition refers to the dropping out of 
subjects from the study. To maintain the study's iritegrtty , this dropping 
out must be random. The subjects must not all ha\e a particular charac- 
teristic. If the dropouts ha\e a parttculai characteristic, like low interest in 
reading, the attrition could influence the dependent variable. The reason 
is that subjects remaining in the study might then ha\e different charac- 
teristics than the original subjects idendfied for study. For example, 
suppose we were studying long-term growth in vocabulary. Over the 
years, we could lose a number of socially disadvantaged pupils who might 
come from non-hoineowning families who tend to move more often. 
Bee ause of a number of factors in their backgromids, such pupils also 
probably show slower vocabulary growth. Loss of these pupils could lead 
to a biased growth curve for vocabulary, that is, the curve might reflect the 
growth only of more privileged pupils. Among piocedures foi dealing 
with differential subject attriuon as a threat are: 
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• CIkkjsc subjects who can Ix; expcciccl to be more stable o\ei time, ciii e- 
fully (lescni)iiig tlie cliaracteustits of those subjects and limiting 
generalizations to subjecis wlio have those eha i n ct eristics. 

• Keep rect)uls about the ciiai act eristics of ihe dropouts on all vai iables 
lelated to the dependent variable. B> judgmental and .stati.stical anal)ses 
.show that ti.e char.ictei istics of the group do iu)t change with time. Or* if 
the groups characteristics change, descril)e those eharatteiistits and 
limit gcneiaii/ations to populations uho have tho.se characteristics. 

Selection interacting xvith other variables. VVIicii groups arc used, 
selcclion may inlciact with all of the above threats to validit). That is, in 
contrast to the other groups, one group ma> experience a different 
history, show different psychological changes, respond differently to the 
testing, I eacl differently to changes U) tests, manifest a different form of 
statistical regression, oi have a different an^ount of attrition. For example, 
let's look at diffei ential attrition. Suppose, we were investigating the role 
of intelligence in giowth in word-meaning skills and comprehension and 
interpretation skills. One approach could involve our sttidying intellectu- 
ally retarded adolescents and intellectually normal adolescents. If, during 
the study, the moie intelligent retarded subjects left school to go into 
vtjcational training progiains, the less intelligent retarded subjects would 
be left in the sttidy. In this case, diffei eiices in the reading skills between 
retarded and noi iiial subjects might be spin iousl> largei than might exist 
in the population. Dealing with selection interacting with other threats 
involves procedures like the follow^ing: 

• L:.se in cxacti) the same \\a> loi eacii group the procediu es cited above 
for dealing with histor>. ps>ciiologicaI changes, respon.se to testing, 
ihanges ni tcsis. statislual legicssion. and attiition eipiaiing groups on 
subjeit chaiactcristics and all experiences which could influence the 
dependent variable. 

• Collet t and present e\ idence that the groups do noi diffei t»n an) ofthese 
thieats to \a!idit\. If thev do dillei , de.scribe how and limit generaliza- 
tions airordingly. 

CunclusunL To repeat. When we interpret longittidinal research 
in reading, weshotdd consider livcd hypotheses about variables like the 
above thieats tov.didit) and identify whethei they are plausible, Ifwecan 
piesent reasons for i ejecting these rival hypotheses, we can ascribe 
changes in the dependent vaiiable to the paitictilar variables we used as 
independent variables in the sti,d>. However, rejection of these rival 
hypotheses is difllcult. We could get dtscotiiaged but we shotild not. 
Realistically, about the most we can ask of ourselves and others is that we 
de.sigii reasonably clean i escai ch and iben limit generalizations systemati- 
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cally <ancl explicitly b> showing which ofthese rival hypotheses might h.ive 
operated. Campbell and Stanley (19G3) said it well: 



The average student or jwtential researcher reading (a!x)Ut the 
threats to \alidityl probabi) ends up with more things to worr) about 
in designing an ex|>erittietit than he had in nund to bcgitt with. This is 
ail to the good if it leads to thedesign and executioti of better experi- 
ments and to more ciicuttispection in drawing itiferetices from re- 
sults. It is, however, an unwanted side effect if it creates a feeling of 
ho|)elessness w ith regard to achiev itig experittietital control and leads 
to the abandonment of such efforts in favor of even more infoimal 
methods of investigation. Further, this formidable list of sources of 
hivalidity ttiighi, with even more likelihood, reduce willingness to 
undertake designs in which from the \ery outset itcan be seen that full 
control is lacking. Such an effect would Ik; the opposite of what is 
intended. 

From the standpoitit of the final ititerpretation of an experi- 
ment and the attempt to fit it itiio the developing science, every 
experiment is imperfect. What a check list of validity criteria can do is 
to make .in expei imentei more aw are of the residual imperfections in 
his design M) that on the relevant points he can be aware of competing 
interpretations of his data. He should, of course, design the very best 
experiment which the situation makes possible. He should deliber- 
ate!) seek out those artificial and natutal labonitories which provide 
the Ik'si opportunities for control. But beyotid that he should go 
ahead with exjxrriment and interpretation, full) awareof the pointsof 
which the results are ecjuivocal [p. 204]. 



Desig)is for studying changes xoith time 

Research designs for longitudinal studies can be a bit elaborate and 
complex. In this section and the next, some simplified illustrations are 
presented for the research (|iiesdons presented earlier. Control variables 
are not consideied hei e because they differ depending on the nattire of 
the problem. In the presentation of de.signs, eHpsis (, . ,) is used to show 
that material exists but is omitted for convenience. Analysis of variance 
procedures are used in the description of data processing. This choice was 
for ease in presentation. In actual situations, the nature of the particular 
problem, the data, or the situation might make more appropriate one of 
the other statistical procedures alluded to al>ove. 

Questions about changes with time pertain to how pupils' behavior 
changes with organic and physiological development accompanying age 
or how their Ix'havior changes as the) participate in common environ- 
mental experiences, like schooling, over time. 
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The Time Design 

Does pupil behavior change over a period of lime? The independent 
v.iri.ible here is time. The dependent variable is the behavior which is 
being ex.imined for the influence of lime s concomitants. To answer the 
question, the beginning time, time jxjriods. and the ending time to be 
worked with are specified. So is the measurement to be used to sample the 
dependent variable. A measurement is made at the beginning time and at 
each succeeding time period. Each measurement is made with the same or 
equivalent procedures. The design can be di.igramed as in Figure 2. 

First Time Second Time Third Time Final Time 

Mcasurcmcm Measurement Measurement Final 

^2 Measurement 



Figure 2. Time 

With this design, plans should be m.ide to deal with five possible 
threats to internal validity which v\ould lead to rival hypotheses in in- 
terpi etation, these ai c history , psychological changes, response to testing, 
changes in tests, and differential subject attrition. 

The data obtained in this design may be processed with an analysis 
of variance model suitable for repeated measures. The summary table for 
the analysis would look like Table 1, 



Table 1. 



Source of Vari?tion 


(If 


Sum of Squares 


Mean Square F 


Total 


N-1 


SSti 




Time 


1-1 


SSTi 


MSt MSTiA^STiS 


Subjects 


s-l 


SSs 


MSs 


Time X Subjects 




SSTiS 


MSris 



The research question v\as: Does pupil behavior change over a 
perivKl of time? The answer is in the F-ratio associated v\ith time. If the F 
vxcie .lot significant. v\ e could infer that a change in behavior did not take 
place ovei the time periods If the F v\ere significant, v\e could infer that a 
change of behav ioi did occur v\ ith time. Then. v\ e could go on to analyze 
the nature of the trend, to see if it were linear, quadratic, and so on. 

In interpretation, wc v\oiil(l v\eigh an hypothesis about the effect of 
the independent vaiiable, time, against lival hypotheses about history. 
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psychological changes, i cbponsc to tcbling, changes in tests, and differen- 
tial subject attrition. If wc could reject these rival hypotheses, then given 
significant statistics we could infer that the changes in the dependent 
variable are attributable to the physical, mental, and/or experiential 
changes which occur with time. 

An example of the time question specified in reading is; In the 
elementary school years, what is the pattern and level of attainment in 
sight vocabular) growth? The research design would involve sampling 
pupils' sight vocabular) at Grade 1 and in subsequent years, at Gmde 2, 
Grade 3, Gr.ade 4, and so on. 

The Time x Groups Design 

Do different groups of pupih show dfferent patterns of behavior change 
over a period if time? The independent v.u iables are time <uk1 whatever 
variable is used in defining groups. To answer the question, the groups 
ai e identified. These groups are different on their defining characteristics 
and the same oi equivalent on all characteristics and experiences related 
to the dependent variable. Then the Time Desigfi is used in exactly the 
same way with each groiq). Figure 3 shows the diagram. 

TIME 

First Time Second Time 'rhird Time . . . Final 'fime 

Group I Measurement Measurement Measurement . . . Final 

§• ^2 if3 Measurement 

£ 

O Group 2 Mea<surcmciit N!casurement Measurement .'. . Final 

J \ «2 ?s3 Measurement 

Figure 3. Time. 



With this design, the threats to internal validity to be planned for 
are histor) and all of the othei threats eithei affecting both groups the 
same way or operating in interaction with selection. 

The data could he processed by an<il)sis of variance model suitable 
foi one independent variable and foi one v<uial)le involving repeated 
measures. The summary for tlie analysis is shown in Table 2. 

The research (|uestion was: Do different groups of pupils show 
different patterns of change over a period of time? We look first foc the 
answei to the leseaich (|uestion in the F ratio associated with the Time x 
Groups inteiactit)n. If it were significant, we could infei that the groups 
showed different patterns of behavioi change with time. Then, we would 
go on to analyze the shape of each group's curve. For example, one group 
nught show a faslei rate of grow {\\ than anothei , one group might show a 
lineal trend indictting a stiaight line development while another might 
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Table 2. 



Source of Variation 


df 


Sum of Squares 


Main Squares 


F 


Total 


t(n.l) 


SSto 






Bctvvccii Subjects 




SSk'C' 






Groups 


g-i 




MSG 


MSG/MSE(b) 


error (b) 


N.g 


SSE(b) 


MSE(b) 




Within Subjects 


N(Ti.I) 








Time 


ti-1 


SSxi 


MSfi 


MSti/MSecvv) 


Time X Groups 


(ti.l)(g.l) 




MSxiG 


MSx,g/MSe(w) 


error (w) 


(ti.|)(N.g) 




MSe(\v) 





show .a qiitadratic trend indicating growth followed b) a plateau or vice 
versa. There are several |X)ssibilities for such differential pauerns. 

If the Time x Groups interaction were not significant, then we 
would look at the effects of Groups alone and Time alone. 

• If bulh the T'imc effect and the Groups effect ucre significant, then we 
couldinfci that the gnnip.s differed in lev el ofresponse.llial change uitli 
time occur! ed, and tliat the groups showed the same pattern of change 
uiih time. I'hen. ue uould go on to examine the nature of llie difference 
beiueeii the groups and the nature of the change uiih linic that both 
showed. 

• If the Time eff ect ueie significant and the Grotips ef feet ueic not, ihen 
ue could infer that the groups did not diffei in level of response, that 
change with lime occurred, and that bodi groups showed ihc same 
paiiern of change with time. 

• If lite Time effeu were noi significant and ihe Groups effect uere, then 
we could iufei that I he groups differed in level of response but that they 
did not sliow a change over lime. Then, we would go on to examine ilie 
lx;lw'een groups difference. 

• If neithei ilie Time ef feci iioi the Griuips effect wcie sigmhcani,tlien we 
would infci thai the gioups did noi diffei in level of i espouse and that 
they did not show- a ch:inge in behavior wiih lime- 
In interpretation, we would need to weigh hypotheses about the 

effects of the independent variables, time and the grouping variable, 
against the lival hypotheses about the seveial thi eats to validity singly and 
in interaction with selection. If wecuuld leject these i i\al hypotheses, then 
we could ascrilx' patterns and differences in dependent variable to the 
independent vaiiables in a way consistent with the results of the analyses. 

An example of a Time x Groups question with implications foi 
leading is. Between the ages of two and six do intellectually letarded and 
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mtellcctuall) lUMinal pupils differ in the amoiim and pattern of change in 
the prefixes, inflectional .suffixes, and deri\ational suffixes they use cor- 
recti) in their oral language? Here retarded cUid normal pupils who are 
alike on characteristics cUid experiences related to language are identified. 
Then, measurements are made of their use of morphemes at ca2, and, in 
subsequent years, at ca3, (:a4, ca5, and ca6. 

Designs for studying changes 
until time and treatments 

Questions about changes with time and treatments pertain to how 
pupils' behavioi changes with some trainingpiocedure they participate in 
over time. 

The Time x Treatment Design 

Docs a given treatment affect pupil behavior over a period of time? The 
independent \ariables are time and treatment. The dependent \ariable is 
the bcha\ioi examined for the influence of time and the treatment. The 
|X)ol of subjects is subdi\ided into equated treatment and contiol groups. 
Given the ti eatnient and control groups, the Time Design is used exactly 
the same way with the two groups. In addition, the treatment group is 
given die treatment, the conn ol group is given the parallel activity. These 
treatment and ctmtrol acti\iiies are started after one or more initial mea- 
surements are taken. The diagram is shown in Figure 4. 



Firj>t Time Second Time Third Time , , . Final Time 

(Treatment iliroughout the time period 
starting after Measurement ^\) 

Treatment 

Group Measurement N!easuremcnt Measurement Fina! 

c "I «3 ... Measurement 

(Control activity ihroughout the time period 
starting after Measurement «1) 

Control 

Group Measurement Nfeasurcment N'tcasuremcnt Final - 

^'2 :f'i ...Measurement 



Figure -f. Time 



In this design, plans need to be made to deal withal! of the threats 
to internal validity , singly and in interaction with selection, that is, history, 
psychological changes, response to testing, and so on. 
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The data could be pKKCSsed by an analysis of variance model with 
provision for one independent variable and one repeated measure. The 
summary is shown in Table 3. 



Table 3. 



Source of Variation 


df 


Slim of S<4uares 


Mean Squares 


F 


Total 


ti(N-l) 


SSto 






Between Subjects 


N-l 








Treatment 


tr-l 


SSfr 


NJSxr 


MSTr/MSEb 


error (b) 


N-tr 


SSE(b) 


NJSE(b) 




Within Subjects 


N(ti.|) 


SSwS 






Time 


ti-1 


ss-ri 


MSt» 


MSTi/MSE(w) 


Time X Tre.iimeni 


(lM)(ir-l) 


SSfiTr 


MSTiTr 


MSTiTrA\^SE(w) 


error (w) 


(ti.|)(N-tr) 


SSe(w) 


^^SE(\v) 





The research question was: Does a given treatment affect pupil 
behavior o\er a period of time? The evidence for the question starts with 
the F ratio for die Time x Treatment iuteracdon. I f the Time x Treatment 
F ratio were significant, we would infer that the treatment group and the 
control group showed different patterns of change as time passed. The 
control group would show the change accompanying dme alone and 
common experiences. The treatment group would show the addiuonal 
change accompanying the treatment. Next, we would go on to describe 
the nature of the change, whether it was equal increments, increments 
increasing in size, or increments decreasing in size. 

If the Time x Treatment interaction were not significant, there are 
several possibiliues. 

• Ifbolhthc rime Fiatio and the ricaimem F ratio were significant, then 
wecouklinfei that both TrcatniLMii and Coniiolgnmpsshowedilicsaine 
pailei a of change w ith time even though the) operated at different levels 
of altaininenl. Next, we wotild go on to examine uhedier the difference 
between the ireainienl and coniiul grotips were in the expected direc- 
tion, that is, if the ireaimeni hnd the effect anticipated. 

• If the I niie F ratio were significant and the Treatment F ratio were not, 
we would mfer tliai the tieaiinenl had no effect and that all subjects 
showed a change with time. 

• If the lime F ratio were notsignificant and the Treatment F ratio were, 
we wotild infei iliai the stibjecls did not change ovei lime but that the 
gioups diffeied. Next, we would go on to examine if the treatment had 
the expected effect, tlun is, if tlie treatment was effective. 
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• if neiihci the Tiinc V latio nor the TrcMtnieiu F vdtio were significant, 
then we would infer thai the treaimem did not ha\e an effect and that 
behavior did not change with time. 

In interpretation, we would weigh hypotheses about the indepen- 
dent variables, time and treatments, and ri\al hypotheses about the 
threats to interiial \alidit> singly and in interaction with selection. If we 
could rule out the rival hypotheses, we could attribute patterns and 
differences in the dependent variable to the independent variables in a 
way consistent with the results of the analyses. 

An example of the Time x Treatment question in reading is: In 
terms of phonics skills, what pattern and level of attainment do pupils 
show if the phonics program in their basal readers is supplemented by a 
phonics program like the Phonovisiial materials in the primary grades? 
The research design would involve identifying treatment subjects and 
control subjects who are similar on all characteristics and experiences 
related to phonics. Theii facility in dealing with the various aspects of the 
vouel and consonant sounds would be measured at the beginning of die 
first, second, third, and fourth grades. Treatment subjects would be 
taught with a bas.n reader program and the Phonovisual program. Con- 
trol subjects would be t*iught with the same basal reader program. In die 
time allotted to the Phonovisual progiam in the Treatment gioup, they 
would be given a parallel activity. 

The Time x Treatments x Groups Design 

Do different groups respond differenll) la a given treatment over a penod of 
time? The independent variables are time, the defining characteristics 
for groups, and the treatment. The dependent variable is the behavior 
studied for influence. The design is a combination of the Time x Treat- 
ments Design and the Time x Groups Design. In summary. The groups 
are identified so that they are different on the defining characteristics and 
equivalent on all other v ariables and experiences related to the dependent 
variable. The groups are subdivided into treatment and control sub- 
groups in a way that they are ec|uivalent on all variables related to the 
dependent variable. The treatment is specified. So is the control activity. 
The measure to sample the dependent variable is identified. The begin- 
ning time, time periods, and final time are chosen. The treatments are 
given to the treatment subgroups over the time period, die control ac- 
tivities to the control subgroups. The same measurements are taken at the 
same time on all subjects. The design is shown in Figure 5. 

In this design, all of the threats to internal validity could operate 
singly and in combination widi selection. Plans need to be made to cir- 



72 



Cofuiucting longitudinal research 



First Time Second Time Third Time . . . Final Time 



(Treatment throughout the time period 
starting after Measuremont x\) 



Treatment 
Subgroup 




Final 
Measurement 



Croup 



Control 
Subgroup 



(Control activity through the time period 
starting after Measurement 

Measurement Measurement Measurement 



Final 
Measurement 



(Treatment throughout the time period 
starting after Measurement 



Treatment 
Subgroup 



Measurement 



Measurement Measurement 

it2 ^3 



Final 
Measurement 



Group 



(Control activity throughout the time period 
starting after Measurement i^l) 



Control 
Subgroup 



Measurement Measurement Measurement 
n\ P2 



Final 
Measurement 



Figure 5. Time 

ciimvent these threalh ur, if that is impassible, to gel information about the 
extent they operate. 

The data could be processed with an analysis of variance model . 
with the provision for two indeoendent variables and one repeated mea- 
sure. The summary is shown in Table 4. 

The research question was; Do different groups respond differ- 
ently to a given treatment ovei a period of time? The research question 
would begin to be answered b> the 7 ratio forTimex Groups x Treatment 
interaction. If it were significant, \^e would infer that the groups re- 
sponded differently to the treatment as time passed. Then, we would go 
on to explore the nature of this differential response. There could be 
many patterns. Fur example, one group might show a steady Ix^nefit over 
the time span while the othci might show an initial benefit followed by a 
leveling off, and so on. 

If the Time x Groups x Treaunent interaction F ratio were not 
significant, we could infer that the groups showed a similar response to 
the treatinentsas time passed. Then, we would go on to explore the nature 
of this similar response. First, we would examine the Time x Groups 
interaction and the Time x Treatment interaction as described above with 
their respective designs. If these interactions were not significant, we 
would look at the influence of Time as descril^ed above with the Time 
Design while looking at the Between Subjects effects. 



BUKKANDAI.I.KN 

ERIC 



73 



67 



Table 4. 



Source of Variation 


df 


Sum of Squares 


Mean Squares 


F 
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• If the Time main cffcu and I he Groups x Treatment interaction effect 
ucre .significant. \\e uonid infei tliat the gioups reacted differential!) to 
the treatment \s\{\\ one sliouing a gi eater difference over its control 
giotip tlian tlie othei and tliai this differential difference continued 
over the lime period studied. 

• If the Time. Groups, and Treatments main effects \\ere significant and 
the Gionps \ T'reatment interaction effect \\ere not, then \\e could infer 
that the Treatment had an influence and that this innuence \\as similar 
for both groups over the lime span. 

• If onl) the l ime and Groups main effects or onl) the Time and Treat- 
ment main effects uere significant) then we uould inlerpiel them as we 
did the main effects in the Time x Group design and the Time x 
T reatment designs » re.spectively. 

In interpretation, we uould weigh hypotheses about the time, 
treatments, and grutips independent variables against the rival hypoth- 
eses about history, and so on. If the rival hypotheses could be discounted, 
then we could ascribe the changes in the dependent variable to the 
independent v. uiables in a way consistent with the results of the analyses. 

An example of the Time x Treatment x Groups question in read- 
ing is: In terms of comprehension and interpretation skills, do pupils 
from non-English speaking backgrounds and pupils from English speak- 
ing backgiounds lespond diffeienti) to supplementing their basal read- 
ing program with a prose listening prognun over the elementary school 
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years? The research dcMgn would involve identifying pupils from an 
English speaking background and a non-English speaking background 
and subdividing them into treatment and control subgroups. These 
groups and subgroups would be equivalent on variables and experiences 
related to learning the comprehension and interpretation skills. At the 
same time, the dependent variables, the comprehension and interpreta- 
tion skills to be studied, would be specified and methods of measurement 
id**ntified. The treatment subgroups would be given the supplementary 
listening program. In the same time period, the control subgroups would 
be given a parallel activity. The measurements would be administered in 
thesam. y to all subjects at the beginning of grades 1,2,3,4,5, 6, and 7. 



Extensions 

We have looked at only a few aspects of longitudinal research. The 
following points should be noted very carefully: 

• ThcMjiicsiions and designs wc have considered are very iincomplex ones. 
Longitudinal research can, and usually docs, involve measures of more 
than one dependent variable, more than one independent subject vari- 
able, more than one treatment variable, more than one set of groups, 
moie than one i>et of treatments, and so on. Also, it can involve various 
proceduies foi identifying subjects and forming groups. It can involve 
various approaches to defining and specifying the independent variables 
and sampling the dependeiu variables. 

• We have alluded only oblit|uely to psychometric and statistical problems 
in longitudinal, oi more strictly time series, research. 'I'hey can be for- 
midable, 

• We did not consider logistics at all. Keeping up with a set of subjects over 
an extended time period involves effort devoted to logistics in addition to 
the logistical efforts vc{|iiired for other kinds of research. 

• We started with tjuestions a.s given and focused on methodology. We did 
not discuss the origin of longitudinal research (juestioiis, as with all other 
research (jucstions, in theory or in othei rational ba.ses. That is, the 
questions and expectations posed must follow from some set of guiding 
principles tiom substantive concerns in reading pedagogy, child 
psychology, learning, sociology, and so on. 

Conducting and evaluaung longitudinal research requires knowl- 
edge about these matters. This knowledge can be ac(|uired in courses in 
substantive areas and research methodology, apprenticeship in on-going 
longitudinal studies, and reference works like Annastasi (1968), Best 
(1970), Cronbach (1970), Edwards (1968), Harris (1963), Travers (1969, 
1973), and Weiner (1971). 
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Conducting hugitttdinal research 



Interpreting the findings: some cautions 



ALHP:RTJ. HARRIS, Emeritus 
City University of Nnv York 



The careful shopper in today's continuing inflation cannot ah\ays 
get the best possible \alue for his money, but he can, if he wishes, subscribe 
to Comumer'i Report and be guided b) the results of impartial tests of 
competing products, such as cars, electrical appliances, and cameras. The 
U.S. Bureau of Standards and the Food and Drug Administration set 
mhiimuni quality standards for a wide \ariety of products. We are pro- 
tected by \arious go\ernmental agencies against dangerous or ineffecti\e 
drugs, noxious pesticides, and contaminated food. 

But no agency protects the educational consumer from practices 
diat are based on shoddy research, or on tradition unsupported by re- 
search, or from inno\ati\e practices that recei\e sensational widespread 
publicity on little othei than unsubstantiated claims. The recom- 
mendations of hundieds of professors, the curriculum guides used in 
thousands of schools, the plans of hundreds of thousands of teachers, and 
the li\es of millions of children can be and ha\e been profoundly 
influenced by pai ticular lesearch studies. As Da\id Russell (1961) pointed 
out, some i eading \ eseai ch has made a real impact on reading instruction, 
and as I larry Singer showed (1970), some good reading research has not 
had the influence on practice that it deser\ed, while other studies have 
influenced practice far beyond what was warranted. 

The importance of reading the full report 

The research literature related to reading is now so \oluminons 
that no single indi\idual can lead it all. The wise consumer of reading 
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rcscMich clocb not atlciiipi tube uniiiivorous. It is no longer possible foi <i 
William S. Gia) to pcisoiially huininaii/.c and re\ic\\ the entire field each 
) ear. E\ en w hen the task is shared by four people and their assistants, as in 
the recent annual suniinai ies in tlie Rimting Risivrdi QuarUrly, thej\>b is 
arduous. 

One is tempted, therefore, to rely on brief abstracts and sum- 
maries. The annual summary in Reading RcMatdi Quartetly, Psychological 
Ahha(L\, and the KRic. system's Reseanh in Eduiation pn>vide abstracts that 
indicate what a stud) was about and often give the main conclusions. 
These abstracts aie non-e\aluati\e and usually dt) not provide enough 
information to allow the reader to judge the validity of the resuhs and 
conclusions. The abstiacts of doctoral dissertations published in Disserta- 
tion Abstta(t.s aie inoie variable. Some of them contain entnigh detail to 
allow a fairl) adet|uate evaluation. Others are so uninfoimative that one is 
tempted to ignore the stud) on the grounds that if the author couldn't 
write a Ix^ttei abstiact, many flaws would be present in his dissertation, 

"Hie propel use of an ab.stract is to provide a full citation and 
enough information to judge if the paper is lelevant to the topic one is 
investigating. If it is relevant, one should try to find and read the report. 

There is a similai relatii>nship betw een a brief journal article w hich 
suminai i/.es a long unpublished lepoi t, and the full i eport, A case in point 
is the Cooffetativi First Gnute Studies, Most people w ho ai e acquainted w ith 
the lesults have lelied on thesuininaiies pi intecl mThe Reading Tea( het . A 
smallei nuinbei have lead the leportsof the Coordinating Center (Bond 
and D)kstra, 1967; D)kstia, 1968) which were published in Reading 
Research Qjnarterly. Even for the director of one of the 27 studies, wading 
through the full leports of the othei 26 piojects was an onerous task. Yet 
onl) in that wa) could one get at the fmc points of methodolog) , statistical 
method, and logical infei ence or could one judge the effects of the many 
compromises with Ideal leseaich design that had to be made because of 
cireumstances in the schools. 

Is ihv sludy centered on one at more clear and teslahle hypotheses? 

This i.ssue has lx.'en discussed in other papers in this collection, 
paiticularl) b) Dr. Wardiop, and will not be dwelt on hei e. In leseaich, an 
h)pothesis is tenable until it is disproved. It is up to the lesearcher to 
determine if the null h)pothesis holds oi not, with legaid to particulai 
effects or diffeiences. I'he finding of a statistically significant difference 
or effect does not establish the variable oi hypothesis in which the re- 
searcher is interested as the cause of that difference. Narrowing the 
possibilities down to one may i ei|uii e setting up and testing a number of 
alternative hypotheses. 




Cautious m nitrrprethig fttuiings 



DoCsS the researcher seem to have been impartial'? 



Most reading research is conducted by people who are not known 
to ha\e a personal stake in the outcome. When reading a repo' t involving 
the evaluation of a proceduie which the report author has de ised, or to 
which he is known to be (juite favorably inclined, the reader should read 
with greater than usual caution. 

Let us assume that five comparisons of a particular beginning 
reading program wiUi other programs have been completed. Two proj- 
ects re|X)rt significant differences in favor of the new procedure, while 
three do not. The two projects with favorable results were conducted by 
the coauthors of the procedure, while the three other projects were 
conducted by people with no obvious stake in the outcome. Under such 
circumstances, one is justified in concluding that the favorable results 
obtained by the coauthors are probably not generalizable to other staffs 
and pupil populations. 

It is not necessary to assume any intenuonal dishonesty or doctor- 
ing of results. In testing the value of a new medicine, it has been shown 
over and over that if the people who administer the drugs know which 
patients gel die new drug and which get a control pill, that awareness by 
itself can bias the results. Valid results in such studies require the use of a 
double-blind procedure, in which the medical staff as well as Uie patients 
do not kiiuu which patients get the real drug. Otherwise, subUe differ- 
ences in mannei seem to provide a stronger suggestion of cure to diose 
receiving the new drug and this influences the results. 

If an educational researcher has a strong bias in favor of one of the 
variables being compared, this bias is likely to exert influence on par- 
ticipating teachers despite honest efforts to carry on the study in an 
impartial manner. The teachers using that variable may feel that they are 
on the researcher's side, and they may exert extra effort. Teachers in 
contrasted variables are less likely to exert similar effort, since favorable 
results for theii variable will be a disappointment for the project director. 
Research assistants have even been known, in a few cases, to have falsified 
results so as to get them to agree with the project director's expectations. 

Unless there is evidence of biased procedure, the careful reader 
accepts the results of such a study as valid for the particular population of 
teachers and pupils. Caution recjuires questioning the degree to which 
similai i esults are obtainable with other teachers and pupils, in situations 
in which a neutral, objective attitude about the variable may be assumed. 

Sometimes an experimenter reports results in w hich his admittedly 
preferred variable fails to show any advantage. This certainly testifies to a 
strong effort to be objective. On the other hand, it does not eliminate the 
possibility that under li uly impartial conditions, there might have been a 
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significam difTcreiicc against that \ariablc. Replication b> other in\eb- 
tigators with other populations ib the tnil) real protection aganiit possibly 
biased results. 

Are the variables clearly and adequately described^ 

Educaiion.il rese.irch reports are fre(|uently restricted in length by 
the policies of thejomnals to which they are submitted. Often a project on 
which die full report ran to aS many as 300 pages has been briefly 
suininari/.ed in an article of four to eight pages. In such a summary, it is 
usually impossible to do much more th.m present die i.s.sues and the major 
results. The reader often finds a gener.il <ind more oi less \ <igue descrip- 
tion of the experimental \ai iables. Control \ariables are less likely to be 
ade(|uately described than new oi experimental \ ariables. Often a control 
\ariable is described in a single sentence w hich states that this consisted of 
the usual procedure, leaving the reader to conjecture what that "usual" 
procedure re.illy w <is. There may .ictu.illy h.iv e been no "usual" procedure 
if each control tcacliei was free to teach as he or she pleased. Radier there 
may have been a hodge-podge of varying procedures. 

Wlieii there is no description of a control variable, or .( very in- 
adequate description, one has to wonder also about die possibility of a 
strong Hawthorne Effect. The rescMi chei w ho does not bothei tt) describe 
what the control leachei s did may not lia\ e bodiei ed to find out, let alone 
to provide direction, training, and motivation comparable to diat given 
the experimental teachers. 

If an unsaiisfyingly brief journal article is important to the reader, 
it may be necess.ii y to locate the full unpublished report and to look for 
more complete presentation of essential details lliere. The kric system 
provides access to thousands of unpublished l epoiis. Complete disserta- 
tions may be obtained through University Microfilms. Ideally die full 
leport should describe each variable in sufficient detail to allow anodier 
researcher to replic.iie the study . An unclear or in<idet|u<iie description of 
a V ariable raises the suspicion that a similai hick of precision chai acieri/,ed 
die operation of that variable in the study. If die readei cannot find the 
infoimalion he wants in the icpoi l,a personal leltci to die audioi usually 
elicits a polite and cooperative response. 

Are the variables potentially fmoerful? 

As any i eader of educational research knows, the most fi equendy 
reported finding is a lack of siaustic«illy significant differences. It is actu- 
ally unusual to find a set of lesulls showing a consistent, significant, and 
unei|uivi)cal difference in favor of one of the variables. Rather than agree 
widi some skeptics that it doesn't really matter what teachers do, one 
should inquiie whether or not the variables being compaied were 
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sufficiently cliffcieni and biifficicntly p nverful to have a reasonable 
chance to produce different results. * 

A researcher who is interested in determining the effects of a 
particular kind of interv ention often finds it eas> to assume that if the only 
change made in the education of the subjects is the intioduction of that 
intervention, the inter\ ention is the effective determiner of pupil per- 
formance on the criterion measures. But let us consider a fairly typical 
example. The researcher is interested in finding out the usefulness of a 
series of 20 lessons, of about a half hour each, intended to improve ability 
to comprehend and follow printed directions. The lessons are written at a 
fourdi reader level and are used with two classes of fourth graders, while 
two other classes, equated with these in mean iQand reading level, read 
similar materials without specific guidance in following directions, A 
special test of ability to read to follow directions is used as a pretest with an 
alternate form as a posttest. Can a difference be reasonably expected? 

Here are some of the [X)ssible sources of weakness in the experi- 
mental variable: 

♦ All i>f the pupils ina> ha\c had some instructional guidance in following 
(liieciions in previous grades, making the exixirinienial lessons just a 
small addition to one group's previous iasivuciion. 

♦ I he total instructional lime for the ex|x;rimenial \ariable was 20 half- 
hour lessons, oi 10 hours, Reading of icxilwoks in mat hematics, spelling, 
and so forth may ha\ e in\ol\ ed an a\eragc of 15 minutes a day following 
directions, which in 20 weeks would amount to about 25 hours of un- 
guided practice, not counting similar practice in previous grades. Thus 
the ex|x;rimenial group ma> ha\e received 10 hours of guided practice 
and 20 plus hours of unguidcd practice, while the control group received 
30 plus hours of unguided practice. It would he surprising if the 20 
lessons produced much of a difference. 

♦ The abo\ e estimates of lime ,speni in unguided praciicc do not allow for 
M>me corrective guidance in reading numlxrr problems and other guid- 
ance in readmg directions duiing periods not lalxrled as reading instruc- 
tion. 1 he) also do not include practice at home while assembling models, 
baking cookies, working for Scout merit badges, and so forth. 

It seems evident that a study like this does not contrast training in a 
partictilai skill with no training, but rather a litde more guided practice 
with a little more unguided pracdcc, Tlie difference in variables is proba- 
bly not great enough to produce any real difference in results, even if a 
completely valid criterion measure were available. Many studies reporting 
no significant differences have been exercises in futility because the re- 
searcher failed to create enough difference l>etween the experimental 
variable and the opportunities for learning in the rest of the pupils* lives. 
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Have af)/jr()f>nale exfMrimental controls been xised? 



In a paper of this length, it is impossible to go into any detail about 
the many kinds of errors which an imwary or unsophisticated research 
designer ma) unintentionall) commit. The long chaptei b) Campbell and 
Stanle) (1963) in the Handbouk of Reseanh on Teaching deser\es intensi\e 
study b) anyone who wants tv) be an informed e\aluator of educational 
experimental studies. To me theii discussion was in\aluable and I ha\e 
reread parts of it again and again. 

Aside from the adequacy of formal experimental design, the alert 
reader should look foi indications that the dice may ha\e been loaded, 
deliberately oi through unintentional eiror, so as to fa\oi one \aiiable. 

Such fa\ oritism can take many form.s. One is allowing contrasted 
groups that were e(|uated at the Ix^ginning of the study to become un- 
equated before the end of the study. In one of the Cooperative First 
Grade Studies that was continued beyond the fust year, one \ariable lost 
20 percent of its pupil population thiough uon-promi;tion at the end of 
»\e first grade, w hile another \ariable lost only 6 pei cent. It is not surpris- 
ing that the foimci \ aiiable came out ahead of the lattei on second giade 
posttests, K\cn if the gioups were re-et|uated on oiiginal pretest scores, 
the presence of 1 I peicent moie slow learners in one set of classrooms 
may ha\e influenced the teachcis in that \aiiable to Mow down theinstiuc- 
tional pace, which would ha\e reduced oppoi tunities to make superioi 
scores. 

A second source of possible bias lies in the assignment of teachers. 
If teachei volunteers aie assigned to a new method, while teachers from 
the non-\olunteei gioiip are assigned to contiol classes, that in itself 
should raise a suspicion that teachci ability and enthusiasm weie not 
evenly balanced. 

A third problem aiises when the \ariables opeiate in different 
schools OI in diffeient school districts. This is often the case in studies 
which draw compaiisons between methods which have been employed in 
dieir re.specti\e schools foi se\eial yeais. In the Cioopeiati\e Fir.st Grade 
Studies it was found that ei|uating populations foi measured pupil 
abilities and teachci chaiacteiistius was not sufficient, some schools and 
districts had high stoies legaidless of method used, and otheis had low 
Stores regardless of method used, and this was tiue aftei an attempt to 
control foi pupil abilities by using a co\aiiance pioceduie (Bond and 
Dykstra, 1967, Dykstra, 1968). Thus it was possible to attribute diffei- 
ences found to the experimental \aiiables only when the contrasting 
methods operated in the same schools and districts. 

A related problem is the unknown and usually unnieasurable 
influence of the attitudes and l>cha\iois of adniinistiatois, supei\isois, 
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and teacheis 111)1 in\ oKcd in the bind) . These may range from supportive 
and cuuperali\e lu ubsli iicliunislic and aniagunislic. Ravel) can one find 
mention of these possible influences in a l eseaich report. Yet those who 
have conducted research in the real world setting of the schools know that 
the benevolent neutiality which is generally assumed does not always 
exist. This can be a most impoi tant source of erroi , despite the fact that it 
is rarely mentioned or discussed. 

Are the measurement procedures used 
valid for the purposes of the study? 

One of the central problems in ; vuclmg research is the selection of 
tests or other measurement procedures wh.ch are used to measure the 
results. The reliability of educational »ad psychological tests of various 
kinds is relatively easy to establish. Test \alidity is not a constant but varies 
according to the many situations in which a pai ticulai test may be used. If 
one is trying to gauge the value of instruction in syllabication, it makes a 
diffeience whether the test of syllabication employed requiresjust count- 
ing the number of syllables in a word, drawing lines to show where 
syllables begin and end, or stating and applying syllabication rules. If one 
is trying to find out the effect of syllabication skill on word recognition, a 
test of oral pronunciation i>f words in a list may giv e quite different results 
f 1 oni a vocabulai y test in multiple choice form. So the results obtained are 
sometimes predetermined by the criterion measures selected. 

Standardized iv^ading tests, often used as pretests and posttests, 
have come under severe ciiticism in recent years. Criterion-based tests 
often ha\e more apparent face validity, but may Ix? low in reliability and 
may employ aibitiaiy pass-fail standards. Ratings and self-report (jues- 
tionnaires have their own limitations. The careful reader tries to decide 
foi himself whether the choiceof measures used in a study was reasonable 
and relevant to the problem. 

All inadeijuate criterion mcasm e can lead investigators up blind 
alleys foi yeais. Foi example, many studies of lateral dominance and 
leading ha\e used only two categories Tor handedness, left and right. 
SubjecLs w ho seemed intei ineiliate in handedness were forced into the left 
oi righr category, and relationships to reading were usually found to be 
non-Mgnificant. Those studies which employed a third category of mixed 
or incomplete handedness weie more likely to find significantly poorer 
leading in the mixed handedness gimip than in those with established 
handedness (Harris, 1957; Cohen and Glass, 1968). 

Is the Statistical treatment appropriate? 

If one is not fainiliai w ith the merits, limitati(ms, and re(iuirements 
for the many diffeient kind.s of statistical procedures used in educational 
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research, one has to assume that the researcher had sdught competent 
statistical advice and leave it to statistical experts to argue whether or not 
the procedures used were the best that could have been employed. 

One of the important differences between laboratory experimen- 
tation and classroom learning is that children in classrooms are usually 
taught in groups. When the same procedure is used simultaneously with 
all members of a group or a class, the group or class should be the 
statistical unit in statistical analysis. If four classrooms are used for each 
variable, and each class is taught as a group, the number of cases for tests 
of statistical significance is properly four classes rather than 100 pupils. 
With an N of four, a substantial difference is needed to satisfy tests of 
statistical significance; with an N of 1 00. a very small difference may look 
significant. The careful reader should note the author's choice of statisti- 
cal unit and weigh the evidence of statistical significance accordingly. 

A related concern is with the actual size and importance of differ- 
ences found. There is a growing practice of reporting only wheUier or not 
a significant difference was found, without giving the stati,stical findings 
on which that conclusion is based. This is a very unsatisfactory practice in 
that it denies the reader the information he needs in order to check the 
author's conclusions. Somewhat better is the practice of giving results 
entirely in raw scores, so that a reader would have to locate a table of 
norms and look up the derived grade, stanine, or percentile scores in 
order to judge how important die difference was. Sometimes a statistically 
significant difference does not amount to much when translated into 
grade equivalents or other derived scores. A careful reader likes to find 
mean raw scores and derived scores, so he can judge the practical impor- 
tance of the differences found. 

Are the authors conclusion justified by hh results? 

In P ading the section of discussion and conclusions in a research 
report, one should keep three main questions in mind: 

• Does the anthoi pio\i(le a reasonable and lof^ical explanation of the 
results obtained? Sometimes bias slums clearl) in an attempt to make 
non-significant results look significant, or to argncawa) lesnlts which do 
not support tlie author's expectations, If objecti\c test results are non- 
supporti\c. pupil and teachei expressions of satisfaction or dissatisfac- 
tion may l)e played up, or vice ver,sa. 

• Does the author focus on one jxjssihie explanation of his results and 
ignore othei possible explanations that nia) be c»(|uall) plausible? This 
ma) indicate a limited \ision raihei than a bias, but it still can mislead the 
reader. If there is just one reasonable explanation possible, the problem 
may he considered .solved. If there are two or moie interpretations 
possible, one looks for reu)gnition ofdiat fact, and foi recommendation 
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for iuUlitioiiciI biiulics \vhicli iniglii eliminate some of the alternatives. 
Putting all reasonable interpretations in tlie leport, rathei tlian just the 
one which the author happens to favor, is a desirable procedure not 
always followed. 

• Is tlic degi ee to w liidi tlie authoi generalizes Uoiu liis results justified by 
the conditions of the project? At times we have been misled into accept- 
ing sweeping generalisations based on studies which utili/.ed (|uite un- 
rcprcsenlalivc population.s, settings, or both. 

A notable example of generalization far beyond the data is the 
statement that a minimum mental age of 6 years, or sometimes 6V2 years, 
is necessary for a child to be successful in beginning reading* Such state- 
ments were quite commonly found in books on reading instruction for 
alx)ut 30 years, from the 1930s into the 1960s. This notion, as Singer 
(1970) has pointed out, seems to have had its origins in a study by 
Morphett and VVashburne ( 193 1) which received wide attention because 
of Washburne\s prestige as a leader and innovator in the Progressive 
Education movement. Morphett and Washburne actually recommended 
that . . a child would gain considerably in speed of learning if begining 
reading was postponed until the child had attained a mental age of six 
years and six months" on the Detroit First Grade IntelligcMiceTest, At that 
mental age, 78 percent had made satisfactory progress in reading, while 
below it, the percentage succeeding diminished. Note that they did not say 
that such a mental age was essential, but only that learning would proceed 
faster. The study had been conducted in Winnetka, Illinois, a suburban 
community in which the average ly was (juite high and whose schools were 
already pioneering in individualized instruction. 

The many authorities who cited that study during the next 30 years 
as a basis for pronouncements about mental age and reading readiness 
ignored the unrepresentativeness of the school system, the instructional 
methods, the rather exactingly high standard foi passing, and the abilities 
of the pupils, and tended to state that a minimum mental age was a 
re(|uirement for success in beginning reading— a generalization which 
went well beyond the conclusions of the original researchers, ft is quite 
possible that some of these w liters never bothered to read the original 
report critically, but simply borrowed thecitation and interpretation from 
a previous v*riter» Unciitical o\ergenerali/ation from unrepresentative 
lesearch led to the unnecessary ix)stponemenl of beginning reading 
instruction for millions of children. 

What this all adds up to is that the consumer of educational re- 
search needs to utilize the skills of critical reading. He does not trust the 
decisions of editors on w hether or not to publish a particular report. He is 
not willing to accept an author's conclusions at face value but applies his 
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own sophistication in icscaich design, slatistical analysis, and iiifcicntial 
reasoning. When he does this, hediscoveis thai a peifectl) designed and 
executed leseaich stiid> is as rare as a completely unflawed diamond. 
With an understandir^g of the constraints under which educational re- 
search i.s cariied on. paiticularl) in school-based research, he does not 
expect complete pei fection. But he tries io be aware of die man) factors 
whic!i limit geneiali/atitin Innu obtained rebulij), and uses this awareness 
in deciding htm fai lo ti ust dieconchisitnis fornudated b) the leseaichei , 
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Linguistics in reading research 

RONALD WARDHAUGH 
U n wersity of Toro nto 



Linguistics is iiiiportaiU in reading research because serious think- 
ing about language is necessary if we are to continue to obtain useful 
insights into language clev elupnient and reading and any relationship that 
might exist between the two. However, it takes considerable effort to 
acquire linguistic sophistication, becoming proficient in linguistics is no 
easy uisk. A doctoral degree in liiigui.stics takes an average of half ado/en 
years of graduate study and marks only a beginning level of competence 
fur ino.st recipients! Sophistication uf almost any kind is all too often 
missing from the w oi k that is dune on language and reading, even though 
attempts are made to appear sophisticated. The principal attempt, of 
course, is to use current linguistic terminology in reporung on work in 
reading. Unfortunately, the attempt sometimes provides evidence that 
the researchers ha\e stepped far outside their linguistic competence and 
undei standing. 1 1 i., unfortunately , all too easy to recognize the garblings, 
die misunderstandings, and die nonsequituis that characterize such en- 
deavors. Terms are misused, concepts are confused, subtitles are over- 
looked, and straw men are suiffed, beaten, and dispatched. Of course, the 
alt«riiari\c i»s not to attempt to use any, or much, linguistic knowledge at 
all— and Uiis alternative is much more common— and to produce the 
instant museum pieces of which the literature is so full. The real difficulty 
IS finding the productive micldleground— the area in which good linguis- 
tic knowledge tan Ix; used to bring forth a harvest worth gadiering. 

Linguistic knowledge nui.st also be used cautiously in language and 
reading research. But it is not necessarily the case that better research 
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would inevitably Ix* done if sophisticated linguistic concepts were intro- 
duced wholesale into that research. On ihccontiary, the results of**going 
wild" about linguistics could well be disastrous. Lest these statements seem 
wildly contradictory to each other, let me pro\ ide some reasons for giving 
this warning about the sometimes "excessive" consumption of linguistics. 
Linguistics is not ^'unsafe at any speed"— raUier it is unsafe at certain 
speeds, particularly at the speed at which one emphasis gives way to 
another. 

The nature of 'linguistics 

Linguists are concerned with theoretical i.ssues having to do with 
what language in general is, about how individual languages are struc- 
tured (for example, English), <md about how languages change and vary 
over space and ume. They deal in powerful abstracuons and they tend to 
use evidence either to support or to deny claims rather than to deal with 
issues exhaustively. Consequently, linguists do not offer comprehensive 
statements about masses and masses of data: foi example, they do not 
usually attempt to w rite longand exhausdve grammars, instead Uiey offer 
tantalizing theories, generally of a very powerful nature with suggestions 
as to how these theories may be tested this way or that. Quite often one 
counter-example — that i.s, one piece of contradictory evidence — is more 
important than a hundred confirming examples for a linguist because the 
theory must be reformulated. It is this kind of sophisticated concern 
which is most respected in theoretical linguistics. The term theoiy does not 
appear to be used this way in research in reading. 

There hy of course, another level of sophistication in linguistics — 
the level which lakes concepts derived from what can be called "frontier" 
linguistics and deals with (|uantities of data in terms of tho.se concepts. 
This is the level of the textbook presentation of linguistic ideas, a level 
\\hich isinevit^jbly characleii/.ed by a certain amount of lag. In a sense, it is 
concerned with yesterday *s linguistics rather than today's. But it is this 
sophistication rather than the first kind which is probably most usable in 
research in language development and in reading because it has, to some 
extent, stood the test of lime— though possibly briefly. We all know that 
five y ears is a long time in linguistics; certainly the "half life" of a linguistic 
idea is generally no longer than five years! 

We should note, howev er, that this kind of linguisdcs is still consid- 
erably powerful. It does not deal merely uith new labels for old bottles or 
new formalisms for old processes. It deals with new concepts and new 
understandings. This point cannot be ov eremphasized. You will under- 
stand its importance if you can appreciate the significance of the following 
observations. There were fundamental differences between Bloomfleld 
(1933) and Sapir (1921), There are fundamental differences Ixitween 
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Chomsky (1957, 1965, 1972) and such post-Chomskyansas Postal (1971), 
Ross (1967), McCawley (1968), and the UkofTs (G. Uikoff, 1970; R. 
Lakoff, 1968) on the one hand and Fillmore (1968) and Chafe (1970) on 
the other. And Labov's linguistics (1972a, 1972b) is very different again. 
Unless you clearly understand what is different among these linguists as 
well as what is the same in their views, you probably should not try to apply 
what they are saying to research in language development and reading. A 
potential researcher might evaluate his ability to do what I have just said 
and thereby determine his competency to do research that employs lin- 
guistic concepts. 

Linguists also tend to be specialists. They categorize and sub- 
categorize not just into phonology, syntax, and semantics, but into sub- 
categories of each of these, for example, into generative phonology, and 
even intosub-subcategorics. The result is numerous subspecialties. It may 
be extremely difficult to dig out from a subspecialty what might be of 
interest in investigations into phonic abilities or sentence processing, but 
such digging is probably necessary for some researchers at one time or 
another. In recent years investigjUors faced with such difficulties have 
occasionally tried to find refuge in such interdisciplinary havens as 
psycholinguistics and sociolinguistics. A warning is in order about such 
"avoidance" behavior. If linguistics is a branch of cognitive psychology 
(Chomsky's famous— oi notorious— assertion) and if Labov's sociolin- 
guistics is really the only methodologically ju.stifiable linguistics (Labov's 
pioselytizing claim), then psycholinguistics and sociolinguistics may turn 
out to be traps foi the unwary , not havens for the tired. Work passed off as 
psycholinguistic or sociolinguistic in nature may be little more than work 
which had failed to confront real i.ssues of lasting importance. 

Linguistics is pov\erful medicine. It should be taken in small doses 
to avoid the risk of poisoning. But in that respect, it is neither better nor 
worse than statistics. Too many studies are spoiled by over-doses of 
statistics, as fragile or dubious linguistic concepts are subjected to massive 
statistical overkill. Just as it is necessary to caution against the occasional 
useofovei-powerfulideasfiom linguistics in oui research, :»o I must warn 
againM the all too regul.n use of over-powerful statistical procedures 
employed in oidei to lesearch trivia to death. Just because you can count 
something does not mean you have to count. And just because :>omething 
IS language, you do not have to treat it as though it existed in some kind of 
splendid isolation from everything else. 

A few simple examples can be used to demonstrate the potency of 
linguistic medicine. In my classes 1 occasionally ask in a somewhat rhetori- 
cal fashion what linguistics is all alxnit and ansv\ er by saying that its goal is 
the achievement of some undeistandingof whatsoinids, syllables, words, 
and sentences are, and hou languages differ, change, and vary and yet are 
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iiuich ihcsanii* Suiilciilbiii liiiguLslicblakciuimcroiKscourbCbdcMgncd lu- 
make ihcni think piucluciivcl) alxiUi such mailers. Nowhere is ihere a 
coiiipleielv clefinili\e slaieineni cu\eiing a single one uf ihe lerins just 
used and nowhere is ihere such a slaieineni on ihe subject of language 
change, \ ai ialion, oi sameness. I Iowe\ ei , )ou do not get t|uile ihis kind of 
impression from looking al books on ihe language aris and reading in 
which sialemenis claiming lo be defmiii\e do abound. Of course, it is 
. necessarv lodo reseaich in ihe language ai is and reading even ihough we 
lack a compleie undersianding of die basic uniis and processes of lan- 
guage. Bui such reseaich should acknowledge die unceriaiiilies dial exist 
and should eniplov die niosl lompleie know ledge lhat is available, Uiifor- 
lunalel), loo man) leseaichers slill confuse sialemenis aboul sounds and 
lellers; use diclionai ) s) llabicalion rules as though they also cliaraclei i/.ed 
spoken syllabication habits, lely on senience-paising formulas that aie 
patcnti) inadei|uaie, unrcnealing, and even misleading, and are quite 
uncleai about what is meant by such concepts as language, dialect, and 
rule. What we must seek therefore in research in the language arts and 
leading is some lonsideiaiion lhat the leseaicher is linguistically alert. A 
few y eai s ago ilii i e was .1 sei ies of ai tides in Tlw Reading Teaclwt on phonic 
generalizations (Bmmeistei, 1968, Clyniei, 1963, Emails, 1967). rhe.se 
articles questioned a whole lot of assumptions, but not the language 
assumptions. Without this (|uestioiiing, what was the point? It was like 
inspecting a building thoroughly in e\eiy oihei lespecl but delibeialely 
ignoring the foundations! 

Lest it seem that I am overselling the need for linguistics in our 
reseaich, let me offei a furthei caution 01 two about linguistics as a 
discipline. Not only i:> linguistics a theoietically oiiented discipline, it is 
also a lapidly changing one. Such change is appaiently typical of a young 
and vital discipline. It aiises fioni diffeient competing ideas about what 
linguists should be doing, that is, about the i|uestioiis they should be 
asking. Theie is no single paity line one has to follow in linguistics. One 
lesult has been a piofusion of poleinical statements, and outsiders may be 
excused to some extent foi using the existence ol such polemics for not 
getting in\olved. .Vftei all, if linguists cannot agiee among themselves on 
what they should be doing and on what the facts aie, why should outsiders 
listen to linguists at all? Why not say , \\ plague on all y oui housesi" While 
such lejeition is understandable, it may be short-sighted. We should 
remembei that those uiidei standings that unite linguists fai outiuimber 
those that divide them, Vou do not get much pi ogress — and intellectual 
excitemeii!— out of agieeing to agiee all the time. In fact, you do not get 
any \\ hei e woi tli going. It w ould Ixr tin w ise to ignore the linguistic consen- 
sus (;n such mattei s as tlu pi iniaty of speech, the duality design-featui e of 
language, the system.itii tonslitueiit-based natuie of syntax, the concept 
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of significant conlidil, and bu on. Thcbc agreements exist among all the 
brouhaha that gets attention in some of the semipopular accounts of 
linguistics. 

Linguistics, however, is not our only source of knowledge about 
language. Anthropologists, philosophers, and psychologists, for example, 
have all found interesting things to say about language, about the rela- 
tionship of language to patterns of social organization, about the nature of 
linguistic acts (saying something is doing something), and about some of 
the mental processes that seem to be important in language use: remem- 
bering and forgeuing,percei\ingand processin|j,and thinking and imag- 
ining. Today, linguists are also beginning to take an interest in the func- 
tional uses as u ell as the formal properties of language. For a long time the 
only interest that must linguists had was in linguistic form. Now many are 
also interested in language function. Linguistics has become less self- 
centered and autonomous and more aware of other disciplines and 
broader issues. The previou.sly mentioned developments of psycholin- 
guistics and sociohnguistics are best understood as indicators of a 
broadening of the interests of linguists. Today, language is being studied 
"in context." that is, in the context of real language use. Conv ersations are 
being analy/,ed. linguistic variation is being acknowledged, and the pur- 
|X)se of speech acts is being gi\en some of the same attention that has 
previously been resei \cd almost exclusi\ely for the forms of speech. The 
problems inherent in such study arc legion, and little substance has so far 
emerged from current concerns with what Libov has called ''secular 
linguistics" as opposed to \loset" (or theoretical) linguistics. But at last 
language is legarded as dynamic and functional (that is, as something 
living) rather than as static and formal (that is, as something dead, fit only 
for a kind of ritualistic autopsy). 

It would be desirable to see some of these same concerns with 
language as li\ing things carried o\er into oui research in reading. The 
functions of language ha\e long interested researchers in the classroom. 
What has l)een lacking ha\e been the necessary tools for doing good 
le.seaich. Perhaps the cui rent trend in linguistics will be useful in helping 
us hnd suitable tools. But let us beware, since those tools are likely to be 
powerful ones, they will need to be handled with extreme caution. 

Liuguislics and research in reading 

What aie some specific suggestionr, and ca\eats regarding the use 
of linguistics in research in reading? Let me make some brief remarks 
about certain consequences of what I ha\e just said for studies of oral 
language and reading, and for research in general. 

Much is known al>out the development of language in children. 
There is a long histoiy of good w oik and a nuiubei of exciting theoretical 
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|)oj>sil)ilitic.s c\isi in.itcnnnl \\n the couibc of dcNclopnieiU, in pai titular 
in the work of McNeill (1970), Bloom (1970), Slohin (1 97 1), Vygoisky 
(1962), and Piaget (1955). Nor can Skiiniers work (1957) bedibiuissed as 
cavalierly as some linguists have done. A fascinating wealth of issues 
awaits investigation. But the da)s of counting «ne t)\er. H)ptuhese.s are 
needed which lan be properl) tested. Many of these slu)uld concern the 
\arious factors which impinge on language learning — inaini) ps)choK)gi- 
cal and cultural factors hcietofure laigely neglected b) linguists. There 
are lots of different linguistic icie.is aiound; now is the time to test them 
against other ideas and a wide \ariet) of data diawn iunu real language 
use. l^mguage is a product of nature ri/iJ nurtuie. We must consider a// 
the variables if we arc to provide a coherent accouiu of language devel- 
opment, not just the one oi two that oui theoretical oiieiuation reipiires 
us to see as the ci itic.il ones. TUv tendency to dojust that is both a strength 
and a weakness in linguistics— a kind of poweiful nariowinindedness oi 
I ) r eocc u pa t ion r es u 1 1 s . 

One particuliU'l) interesting pioblem has to do with how children 
grow up to Ix* linguisticall) iliflerent fi oin each othei , We must acknow I- 
edge that linguists have long been inteiested in how people are the same, 
not how the) ai ediffei cut— linguists swept vaiiation under the lug lathei 
consistenti). (xMupaie thisemphasisJI you will, with the usual ps)cholog- 
ical emphasis on difleiences, noimal disti ibutions, and ranges of varia- 
tion. Much icseaich in linguistics glosse.s ovei diffeiences and employs 
laigecovei teims togioup subjects togethei. Oiieof the most interesting 
developments in icccnt studiesof child language development has been a 
leturn to intensive stud) of individual children, as in BrowiKS woik 
(1973). foi example, (ihildien studied in this wa) tuin out toshow differ- 
ent developmental p.mcinsand to use language tlifleienti). Peihaps this 
is one aiea th.it could be e.xploied innnediatel), foi such exploiation 
could have rathei interesting educational conseipiences. 

So fai as I eseai ch in i eading isct)ncei ned , one of the gi eatest needs 
is to look at some of the basic linguistic units employed in leading 
instiuition -sounds, s)llablcs, wmds, and sentences. What the) aie and 
how tliev woik in the spoken language is important as aie theii wiitten 
(Oil elates and the natuie of the coi legations. Sounds and spellings, s}lla- 
bles and syllabication sv stems, woids and meanings, sentences and 
sense what a vast aiiav of i.ssues they conjuie npl And what complexitvl 
It is verv eas) to think of topics to lesearch. What are the units and 
piocesses of speech peiception? I low do these con elate with visual units 
<md piocesses?- What does the e)e see? I low does it see? I low is meaning 
conveved? How lineai? How pieilict.ible? How context dependent? Is it 
conveyed the same in wi itingasit is in speaking? Wiitingis not just simpl) 
speech put tlown on p.ipei «rs anvone who h«is evei wiitten an) thing 
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knows. Moieo\ci, the piiipobcs of wiiting — and consci|iientl) of 
reading — arc \cr) difTeicnt iionx ihobc o( bpcakiitg and listening, as 
Vygotsky (1962) obser\ed. What is the function of reading?^! t is a very 
different function from spe»jking — more abstnict, remote, sophisticated, 
specialized, and disjK'nsable. 

Of course, language is only one component in reading instruction. 
The linguistic system is only one system among many, systems of atten- 
tion, perception, meniory, and priKessing are also iniportant. Language 
itself relies on these systems. Reading is also only one kind of 
infoi mation-gathering system a\ailable to people and must be seen in 
relation to other systems. After all, in answei to the abo\e (juestion about 
the function of reading, it may Ix; that foi many petjple reading is a largely 
unnecessary connnunication system. 

A lot of work of high (juality can be done in reading research right 
now iK'cause thecjncepts necessary foi that uoik presently e\ist. But the 
coujplexity of the pioblems is such that healthy caution— even 
.skepticism — is au\isable. The \\hole aiea of phonics instruction needs a 
thoioiigh re\\(jrking. The linguist's notion of linguistic rule is much more 
intere.sting than the leading researchei 's notion of phonic generalization. 
The disco\eries of recent years alxjut English phonology and the hypoth- 
eses about the units and piocesses of phonology and the lelationship of 
speech to uiiting could all beai seiious examination. Children know Uie 
sound .system of theii language. We know something alxnit that knowl- 
edge and we kmm .siHuething about oi thographic principles. We need to 
in\estigate how we can use the child's knowledge of phonology on his 
behalf and how we can demonstiate to him what connections there are 
between I he soiukIs he .speaks and the letters he sees. 

Studies of syntactic de\elopment and of readability can profit from 
the laige ijuantities of woik done on Knglish syntax and .semantics in 
lecent yeais. This wdi k has allowed us to gain nuny insights into just what 
adds to the complexity of Knglish sentences. It is not length alone that 
bi ings about cumplexU) . Length mei cly i ef lects complexity in a gi oss but 
not inaccui ale way . Sentences ai c toniplex bctauseof ihc enibededne.ss of 
clauses within them» Ixrcause of ijuantirieis (woids like wnu, all, an)f </wA) 
and negati\es» becau.se of theii piesuppositions, entailments, and con- 
texts, ami .so on. Counting this and coi i elating that ai c not enough — they 
I epi escnt (juitc ti i\ iai kinds of i e.seai ch acti\ ity . Counting and con elating 
with a computer serves only to glorify triviality. 

Dialect differences tonlinue to piinide anothei source of research 
in>piiation. L'nfoi tunately tluilttt has been a much bandied about teini. 
lndi\i(lual language \arianon is piobably ju.st as impoitant as those few 
things whith imite a few speakcis tt)ct)n.stitute a "dialect" gioup. Much of 
the leseaich piesently done on tlialccts and leading is leally leseaith on 
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SDciocultiual (lillcMciitt'S aiul on Ic.nning bt)lcs. The language tuinpo- 
neiu in buch i eseai cli hab been simplified and aliemiated to bueh an exieiil 
that nianv of the i exults ofiueh i eieai eh should be labeled "unsafe for c .i> 
eonsunici." I would paitieulailj wain about all the eontio\eis\ that has 
swirled baek and foi th on the subject ol ' Blaek Knglish," racial dilTei ences 
in intelligence, the *'ei|ualit)" of all dialects, language dilTeience and 
deficienc), and so oix. The liteiatuie on these topics is full of bigoted 
attacks on bigoli), intoleiant outbuists in defense of loleiance, and |)as- 
sion and propaganda parading a: scholarship. 

A seiious concern foi linguistic issues could open up research in 
leading in ways ahnost undreanied of until a few )eais ago. Whatever 
reading is, it is sui el) a language pi tjcess. Linguistics is the seritms stud) of 
language. If weaiee\ei togainatleai undeistandingof ieading, we must 
inteiest ourselves in the language coniponeni in leading. Linguistics is 
certaiid) oui best bet if we are tiul) seiious in increasing that 
nndei standing and inci eased undei standing is what leseaich itself is all 
about. 
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Linguistically sound research in reading 
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A psycholinguistic vantage point 



Rcscaicli ill leading in ilic final c|uancr of ilic twciuicih century 
lias entered an age of science. A ps)cli()liiigiiistic \aiitage point has 
emerged based on a iiinnber of key premises. 

These premises are simple, soine will say self evident, yet they 
radiciilly reorient and refociis research in reading. 

Reading is now \ iew ed as one of four language prcKesses. Speaking 
and w riling are piodiicti\e. expressive piocesses. Reading, like listening, 
IS a receptive piocess, no less active than wiitiiig, although the ps>cholin- 
giiistic activity is internal and not observable. 

Readers, like listeners, speakers and w liters are users of language. 
Coiiiiiiiiiiication of nieaiiiiig is what language is used for. In productive 
language pi ocesses, speaking and w l itiiig, language is encoded; language 
nseis go from language to meaning. In receptive language processes, 
listening and leading, meaning is decoded, language users gel ineaning 
from language. 

1"lie new scientific premises lecpiire researchers also to see lan- 
guage in its social context. Liiigiiage is both personal and social; it is the 
iiiediinii of coniiiiiiiiicatioii and the main vehicle of human thought and 
learning. 

\Vi iueii language development in human societies comes after oral 
language development, ai die point wlieie coiniiiuiiicatioii must extend 
over lime and space. 

Foi individuals, wiitteii language almost alwa)s also tomes after 
oial language development. It comes at the point where the individual in a 
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literate society lecogni/es the peiboiial need for moving Ixjyoiul face-to- 
face conninmicatioi) to interact with unseen, perhaps unknown writers. 

So far nothing IVe said is either the result of research or profound 
theoretical insights. These premises represent a point of \iew in the most 
literal sense of that phrase. One need oid\ look at reading to see them. 
They are in no need of scientific verification since tliey are so evident— 
that is, provided that one looks directly at reading. 

Conventional wisdom 

Unfortunately many people, including researchers, have not 
Ix'giui consideration of reading from the simple direct vaiuage point that 
would lead to awareness of these premises. The study of language in 
general, in fact, has been plagued by the tendency to proceed from 
iniexamined traditional beliefs about language. Ct)nventioiial wisdom 
alx>m language is so deeply iixned that e\en researchers committed to 
scientific method and logic often plunge into research on language and 
language learning w ith no attempt at consideration of the facts of linguis- 
tic reality. 

The known and the itnknowable 

Many research studies are further hampered by u.se of one of two 
opposite but w idely accepted \ iew s. In the first \ iew . language is treated as 
being so well understood that it does not need examination. This trans- 

late.s to such statements as» "Everybody knows that This view in 

reading is one of the reasons the public is always so susceptible to attacks 
on current reading instruction. Since e\eryl>ody knows that the way to 
teach reading is through phonics, it follows that piogiams that aren't 
phonics programs nuist l)e the work of idiots oi a delibeiate conspiiacy to 
keep kids from learning to read. 

On the other hand it is very popular for people speaking on 
reading to lake iheopiK)site \iew. I^mguagcisan inifathomable mysteiy. 
It is unknowable. This leads to justification of peisonal ignoiance. by 
statements like. "Nolxjd) knows how reading works, therefore . . This 
then Ix'comesa rationale foi trial and error— particulaily in methods and 
materials. If no one knows how reading woiks, anything is worth tiying. 

An obligation to Ix* scientific 

But language pr(x:esses— reading included— are neither univei- 
sally understood noi unknowable. This leads me to my final premise. 
Those conducting research in reading ha\e an obligation to begin on a 
base of scientific insight and understanding. They must mo\e to a clear 
unf)bstructed \antage point in looking at language, and they mu.sthecoine 
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familiar with the best available theory and knowledge. Furthermore, they 
must use tools appropriate to psycholinguistic research. 

Linguistic imigfiLs from theory and research 

Research and scholarly thought have provided a number of in- 
sights about reading which .expand on the base of the premises Fve 
outlined. Both research and theory are in dynamic stages. That means 
that new knowledge is being produced at a rapid rate. It also means that 
there are compeung theories and conflicting findings emerging. This 
condition is notjiistification for ignoring progress or reversion to a golden 
age when things seemed simpler. It is just such a dynamic state of affairs 
that opens up whole new directions in research— provided the research 
community can hang loose and remain open to innovauon. 

Some highly productive insights from scientific language study 
have <>merged in the past decade and a half: 

1. All children develop language competence. This development is so univ- 
ersal and rapid that some scholars have concluded that langJiage is essen- 
tially not learned but innate. What is most remarkable is that children 
at(iuire noiju.si a set of rules for generating new language; they can say 
things ihcyVe never heard. 

2. Language accpiisilion relates to huin.ui need for communication. The 
mechanisms and motivation foi acquisition of language operate in both 
wriuen and oral language. 

3. Language difference is to he expected. Lmguage grows and changes to 
meet the changing needs of its usei s. Difference must never be confused 
with deficiency. Your dialect is not a funny way of speaking mine. 

4. Umguage is learned in the context of its communicative use. Learners 
treat it like concrete learning if it is meaningful, and the meaning is 
relevant and significant to them. Lmguage is only abstract when it is 
fragmented and/or divorced from meaningful use. 

5. Language competence and language I)chavior are not the same. Compe- 
tence results in behavior; it is the control over the process which results in 
behavior. Behavior reflects but is not eciiiivaleni to performance. 

6. l o infer the competence from behavioral indicators, it's usefiil to post- 
ulate a deep language structure and set of rules for generating the observ- 
able surface phenomena. 

7. Research in reading must Ix; process orientccK it must use behavioral 
iiulicau)rs to infer underlying competence. 

8. The human brain is the organ of information processing. As such, it 
directs the eye and ear and makes selective use of its input channels. 
Perception in reading is largely a mailer of what we expect U) see. hi oral 
reading the mouth says wliat the brain directs it losay,noi what the eye 
sees. 
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9 KfTc(li\e ivading is jchicving coherent meaning. Efficient reading gets to 
meaning with the least anionnt ofeffoit neccssar), and uses minimal 
perceptual input. 

10 Language processes, reading included, cannot be usefully studied by 
reducing thenj to manipulation of constituent units such as letters oi 
words or distorting them l)\ looking at them in highly limited and unusual 
circumstances. 

1 1 Readiiig, like all language processes, must Ix; studied in the personal and 
social contexts which give it purpose. 

Even if researchers find it hard to accept some of these things I've 
labeled productive insights, they should still be geneiating exciting re- 
search ifonly in the attempt to reject them or demonstrate an alternative 
explanation of the phenomena involved. 

Popular unenlightened and itnenlightening research practice 

Fve suggested earlier some key reasons why reading researchers 
have not begtin with an awareness of linguistic realtiy. I'd like to explore 
now some reasons intrinsic to popular research practice that might ex- 
plain why so much research in reading is both unenlightened and unen- 
lightening. 

Narrow vision 

Researchers fre(|uently operate within very narrow frameworks. 
They often pluck a small item out of current practice or select a question 
of concern only within a specific program of reading instruction as the 
basis for their research. An example might be to study an experimental 
method of "leaching consonant blends." Such research not only suffers 
from being of value only within the narrow methodological context in 
which such instruction is used but it also suffers from the usual failure to 
examine the relationship of the item selected to the general question of 
children learning to read. 

Restrictive models 

Research on reading instruction has tended to be dominated by 
research design models which aie of limited value in providing useful 
information on teaching and learning. This seems to stem from a desire to 
achieve "rigor" and respectability without a sound theoretical base. Most 
common is the use of the experimentaUonlrol group design. An attempt is 
made to obtain significance, reliability , and validity through careful ma- 
nipulation of data using statistics based on mathematical probability the- 
ory. 

Some pitfalls in the use of experimental methodology are fairly well 
known: 
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• Vai ubles ai c hai ihu t cjiui ol. l oo much is happening in tlassrooins that 
can't [>c monitored and regulated. 

• Contror'treatments" are usually pooi 1> descril)cd and poorly controlled. 

• i;nv\arranted assumptions are made that all experimental classroomsare 
equal and interchangeable as are ah control classrix)ms. 

• It isn't possible lo control learning v\hich Unos place outside of school, 
planned aixfijlM^P^**""^^^- 

• Results of instruction may not show in any measurable sense until long 
after it is receivfed. ''Results' of short term experimental studies are 
t!i ere fore unreliable. 

• Some important aspects of reading are unassessable in an) quantifiable 
sense. 

• Conclusions are usuall) based on perfoimance on norm or criterion 
referenced tests. Such pei foi niance may not adecjuatel) represent gains 
in competence. 

But there are more basic reasons for rejecting the experiiuental 
mo<lel as a basic tool in research on reading and leading instruction. 

At best it can only "prove" or "disprove" a small set of hypotheses 
already believed lo be true. It plows no new ground, provides no new 
insights. 

The requirement for mass data and random samples causes a focus 
only on central tendencies in stiitistics, whereas individual variation and 
deviation may be must enlightening. We count "right" answers instead of 
examining wrong ones. 

Manipulation of data— however rigorously it's conducted — can 
ne\er make up for the original poor quality of the data itself. Sound data 
can only come from a base of knowledge, sound assumptions, and a 
theoretical framework that give the data value. 

Ofwhat valueisitto prove everyone does something if understand- 
ing /iom one person does it is what we really need to know? 

Data worshipping 

Sometimes it seems that research itself becomes confused wididata 
collection and manipulation. There are so many elegant processes avail- 
able, particularly with easy access to packaged computer statistical pro- 
grams, that correlate, regress, factor analyze, and otherwise cause data to 
engage in impressive behavior, that meaningful results are lost. 

Having produced tables, charts, arrays, and matrices, researchers 
engage m ex post daiuvi speculation about why the statistical relationships 
exi.st, what to do about achieving them if they are good, or which should be 
eliminated becau.se they are bad. Judgmentas to whether they are good or 
bad is often made on a common sense level. 
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In reading and reading instruction with multitudes of texts, tests, 
workbooks, readability formulas, and management systems available, it's 
easy to generate great ([uaniities of data'. Making sense of the data, before 
or after statistical manipulation, requires some knowledge of what read- 
ing is and how it is learned. 

Reifying tests 

Tests are always developed to pro\ ide insights into phenomena not 
readily obtainable in the course of ongoing observation, teaching, or 
learning. A test maker builds a test which he hopes will reveal, through 
test performance, the competence he is seeking to examine. Building a 
valid lest requires much more than administering it to \arious standard- 
izaion groups to establish statistical norms as well as statistical reliability 
and validity. Building a test requires a theoretical model of the compe- 
tence to be examined, based on adequate research and knowledge. Sub- 
tests, items, and tasks must relate to reality through this model. This 
requirement is as true for criterion referenced as for norm referenced 
tests. 

When a researcher constructs his own test he is generally held 
responsible for demonstrating the \alidit) both through theoretical and 
Statistical means. But if he uses someone else*s test — particularly one 
published and in wide use— he is absolved of such responsibility. 

Researchers frequently equate performance on a test in reading 
with reading itself. Each subtest— "vocabular> ," "paragraph meaning," 
"word recognition"— is assumed to be a real isolatable coinpetence or 
aspect of a general competence called reading, and research data are 
reported not as test performance but as if the competence itself were 
being measured. Researchers rarely say "The subjects had poor perfor- 
mance on the subtest on paragraph comprehension." They simply say the 
subjects had poor paragraph comprehension. The reading test becomes 
reality itself, and the researcher often does not go beyond test perfor- 
mance in considering the significance of data. Too often in research 
repoi'is reading is performance on reading tests. 

Confusing science and technology 

A contemporary madness results from the assumptions that all 
uses of mechanization, industrial organization, cylxii netics. and other 
aspects of technology are scientific. Technology is treated as a synonym 
for science, it's as if people think that all that was necessary to get humans 
on the moon was to build a rocketship to get them there. Humanity 
couldn't have gotten ihere, as far as present science knows, without one; 
but science made both the ti ip and the needed technology possible. 
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Funhcinioic, thcic die iiifniitc luimbLMS uf Ubesul'spacc lechi\olog> that 
arc possible which are worthless, absurd, or both. 

In reading instruction and reading research, worthless and/or 
absurd uses of technology are frequentl) treated as scientific. Computer 
assisted instruction, computerized data gathering, instructional man- 
agement systems are being built on wholly unscientific assumptions and 
views. They produce neat, inanipulable data which are treated with un- 
warranted respect by researchers, who are awed by their technological 
trappings. 

Technology can be, in fact must be, used to facilitate learning; but 
educational researchers are scientists w ho must control their tools and not 
become controlled by them. Noi can they retreat from their obligation to 
people into the impersonal coinfoir of the machine, the data, or the 
management system. 

RecTcating the world in the laboratory image 

Researchers often create simplified \ersions of phenomena and 
experimental designs in their laboratories in order to study the world or 
some aspect of it which is too complex for direct study. Under special 
experimental conditions they can gain useful insights. The goal is then to 
lest these insights against reality. But in reading and language research 
particularly, there has been a strong tendency to view reality as an exten- 
sion of the laboratory and the narrow experimental view. Phenomena 
isolated for study are treated as unchanged from their occurrence in 
uncontrolled re.ility . This is a problem of great concern w hen researchers 
or others leap fiom lesearch to development of reading methods and 
materials. There is a i eal world, and ideas— howev er clev erly tested in the 
narrow confines of the laboratory— must also be placed in the context of 
this real wx)iid. 

Mindless empiricism 

Objectiv ity in reading research is often construed to mean 'hat only 
the tangible, measuiable, directly observable aspects of things are legiti- 
mate concerns. Values, philosophical positions, theories are viewed as 
unnecessary, subjective, and dangerous. 

What often results is a mindless empiricism of the sort demon- 
strated by the Blind Men of Hindu.stan. "All we know is what we can 
measure" seems to be the motto of these researchers. They are content to 
stay on the sui face of things, to add, but not to synthesize. They particu- 
larly reject consideration of values, In the name of scientific objectivity 
they disdain responsibility for the effects of their studies. No field of 
knowledge has been able to pi ogress without theory to explain observa- 
tional phenomena, togeneiate hypotheses, to predict behavior. Reading 
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knowledge has tended not to ino\e oi build on research because of 
rigorously superficial, atheoretical leseardi uhicli has prevailed. 

Atheoretical research often produces absurd conclusions which 
are apparently supported b> empirical studies and are accepted even 
though they contradict the reality practitioners must deal with. Recent 
testing in California, foi example, appeared to^h<m that disproportion- 
ate numbers of Chicano and black school age children suffered from 
aphasia as compared to other populations. One nmst regard this conclu- 
sion as unlikely if not absurd, if these data are placed in the context of 
what is known about language de\elopment and language and cultural 
difference. In that context, we would do better to reconsider the tests than 
to group children and build curricula based on the results. 

The dissertation process 

Doctoral research ought to pro\ide opportunities for scholars to 
explore the frontiers of reading. Yet one of the most conservative 
nUluences on reading research is the dissertation process. The doctoral 
candidate often is up-to-date in his or hej knowledge of research and 
thought. The student should select a current problem or issue and utilize 
innovative research methodology. He or she ought to know more about 
the topic and metlK.dolog) than the chairnian or the members of the 
committee of three to fue facult) members, each of whom may view 
reading research from a somew hat outdated \ antage point. By the time he 
or she has satisfied a committee and an archaic set of university re<|uire- 
ments. the researcher nja> ha\e been forced to c<mipromise methodol- 
ogy, to touch irrelc\ ant bases, to answ cr already answ ered <|uestions. I f he 
or she hasbnilta background in linguistics and psycholinguistics, commit- 
tee members ma> defend theii ignorance b> belittling the significance of 
the knowledge .insights, or \ antage point being used. Often the effect is to 
push a student back to safe studies with traditional instiuments. Some- 
times the student pei.se\eies, but the .stud) is cluttered and weakened by 
conditions imposed b> committee members. Universities and doctoral 
committees must gi\e .serious diought to opening up the dissertation 
process and liberating students f rom the yoke of tradition. 

The funding process 

Similari). the process b> which research gets funded works in a 
(onservalive manner. Proposals are read and judged by those who have 
made their mark and ha\ e \ ested interests in the status (|uo. Traditions of 
what "good" research proposals look like grow up which make it difficult 
to tell the ingenious inncnator from the ciack pot. Official requests for 
proposals often are written in a language and conceptual framework 
which eliminates ahernate, productive models. 
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Furthenuuic rcseaichcrs nuist "go where the money is" rather 
than deal with research that needs to be done and which they arc compe- 
tent to do. Sometimes research support goes to those who can inchide the 
right timely key words in their proposals. 

Another problem is that money goes to money. New researchers 
with great ideas but no past funding find it hard to get that first grant. 
Universities and funding agencies both need to expand their support of 
high-risk research. 

Tlie research. pecking order 

Researchers like other human beings tend to be infiuenced in their 
attitudes toward others— by their perception of the status of those others 
relative to their own status. So psychologists are in awe of some linguists 
but show disdain for educational researchers and their work. This status- 
conscious view causes some researchers from disciplines other than edu- 
cation to Ix* less careful in doing their homework when begiiuiing re- 
search in reading. Sometimes they don't bother to find out what's already 
known. Sometimes they are unscholarly in the way they loosely interpret 
their data. Sometimes they state opinions authoritatively which have no 
basis in their research, particularly when extrapolating from data to 
methodological applications. Conversely, eductional researchers some- 
times show the up-tight, self-conscious behavior of the low-status group 
member who accept.s the derogatory stereotype of his group. They be- 
come ever more conservative and narrow in theory, methodology, and 
research scope than the "pure" scientists thty emulate. Practical, applied 
research enjoys less status in many disciplines than pure research which 
pursues knowledge for its own sake. 

Research oriented to the solution of real problems— illiteracy for 
example— is treated as unworthy. That view is reflected in promotion, 
salary, and research support policies of universities. It influences accep- 
tance policies of research journals. It has even had effects on federally 
funded programs designed to deal w^th real problems. 

It is becoming increasingly clear that sound research in reading is 
going to 1 equire an interdisciplinary base. But interdisciplinary teams will 
achieve their goals only if they can operate in a climate of mutual respect. 
Elitism in any form has no place in research planning, fimding, or per- 
formance. 

Broad jumps and other leaps 

Reading has been particularly plagued by a tendency for 
"methods" of insti uction and sets of materials for instruction to be built by 
researchers or consinners of research on the basis of single conclusions 
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froin narrow or liiiiiicd resc.ircli. Researchers have the obligation to be 
cautious in generalizing from research conclusions to iinpliratioiis for 
practice. They also need to be careful u hen the> change hats and become 
authors of basal reading texts that diey consider all necessary inputs, 
become informed about all rele\ ant concei ns, and operate \x ith the same 
scientific cautions which they employ in their research. 

Early uses of linguistic concepts in reading research led to die 
so-called "linguistic method" and a rash of "linguistic readers/' all based 
on a single linguistic principle of minimal differences betu een phonemes. 
Phonics programs were renamed "decoding" programs to capitalize on a 
popular misconception that learning to read \x as "decoding" phonemes to 
graphemes Better instruction will result from scientifically based re- 
search but not in a simple direct ua>. Research is needed on how reading 
works, how it is learned, how effective various programs for instruction 
are. The knowledge from such research must then be integrated with 
other practical knowledge to produce more effective instruction and 
more universal learning. 

What are the requirements of good reading research? 

Some of the reasons for poor oi nonproductive research have been 
explored above. In contrast, here are some of the requirements for 
linguistically sound reading research. Simpl> speaking, the more tliats 
known about reading, the more necessary it is for researchers to base their 
studies on a wide understanding of uhat is known. Research must be 
consistent with modern insight into language and language learning. 

Research studies of large numbers of subjects must give way to 
depth suidies of small numbers, such as those popular in linguistics and 
developmental psycliolinguistics. If a researcliei can find through the 
study of a single subject how reading is used to comprehend a writers 
message, an important contribution be made to human knowledge. 

Real people using real language in various real situations must be 
the objects of research if we are to understand reading as it really is. 
Research problems are being generated today from a variety of product- 
ive sources. 



Theories and models 

As we attempt to understand the phenomena observable in read- 
ing accjuisition, models and theories arc eniei ging. These models produce 
predictions and hypothetical explanations of reading phenomena. Very 
useful studies can emerge designed to support oi reject these theories, 
iiKKlels, and hypotheses. Studies u hich pi <n ide data w hich are consistent 
or inconsistent with a theoretical v icw w ill make cleai \\ hedier any existing 
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model i.s most cr.scdil in dealing uith reading. The models become more 
powerful as th^) are tested against lealit) and oiii knowledge of the 
reading process^ grows at the same time. 

Anomolics 

Another base for generating useful research in reading is in the 
numerous unexplained anomolies which now exist. For example: 

• Whv do mhimII) all iliiklicn iKqiiiic oial laiiguagc uillumi prolcssioiial 
assisianc e >i;i some t liildicii don't .sccin to learn to read casil) and well? 

• Why di> sthoids appeal w hv less sincessful in leatliing reading to boys 
than girls, blatks than whites, ixmi kids than rich ones? 1 oo often we 
Man Iroiu ilit- "lati" ol the ilifleientc in atliieveiueni and speculate on 
cause. W e need to understand n hy and in w hat wa\ s we're less successful 
with some groups of kids than others. 

• Whv, 11^ a Inerale sotiel\ widi univeisal attess to instruction, do some 
people x-enmin fnnriionally illiierate? 

RejeciccI absu rdily 

As we come lo midei stand more about the reading pnKess. much 
useful researcFx can come fiom leexauiining the absiud fmdings of less 
enhghtened research. Why do some groups have low norms on IQ and 
achievement tests? (We know it isn't a real difference.) Why has knowl- 
edge of the alphabet been a fair predictor of later performance on 
readmg tests? Why dc)e\en lidiculous instructional programs succeed in 
helping some k.ids to learn to read? 

Unsolved real problems 

Bold nev\ approaches to real problems in building the knowledge 
base necessary to achie\e uni\ersal literac) are possible and necessary as 
the reality coin<;s into focus. 

A key e>cample of an urgent unsoKed problem is how to gel at the 
competence uri del I) iug leading comprehension. In this, as in many other 
questions, it's e-;isici to see what's wrong with current solutions and prac- 
tice than to de V <ilop new approaches. We know that all current techniques 
try U) nder coiiapiehension competence from post reading performance. 
Such inference- is never really adequate. It is always distorted. 

Anothcir majoi unsoKed question is how much the silent reading 
process differs from the nuich more easilj studied oral reading process. 

These c^Tiestions frustrate but they also tantali/e. Probably break- 
throughs will c ome f lom inteidisciplinar) teams, drawing on knowledge 
and methodola>gy fiom linguisucs, psychology, and education. 
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The lime has come 

It is not unreasonable to <isk mnv that researchers bring a nuxlern 
base of knowledge to their research. It is not unrcMsoiiablc to expect them 
toconduct research worth) of time, effort, and CNpeuscwesearch which, 
whether big or small, contributes to some degree to mo\ ement toward the 
goal of universal litenicy. 
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Social aspects of xuorhiug xuith the schools 
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III .itl(lirn)ii U) llic iiioic icthnic.il (Icsign .ind incasiii cniciil prob- 
lems ijf reading iCNC.iicli. llicic aic IIIOIC piaciicalaNiK'US involved in the 
iiuiiagciiiciiloreiiteiliig iiiU). opeialiiig uilliiii. and exiling from school 
.situatioii.s in uliith one cuinliicts leseaith. riii.s ariicle will give examples 
of ilie •'sotial" picjbleiiis ul slIuhA research and provide specific sinie- 
iiieiiis. legaiding geiieiali^uiun.s that have lx:en derived from experi- 
ence, lo aid in coiidiitliiig le.seaich in School .sellings. Fhe comments are 
meant to appK lo lK)lh small Scale (one class) and large scale Iniaii) class) 
research projc*ct,s, 

Euterini( the school 

The reseaichei slionhl letogni/.e lliat educational research is not 
higliK \ahied In the mIiooIs. in fad. it <)rteii is perceived as iiniiiiporiant 
hi 111% opinion, school |H:isoiiiiel lai loo often seem more iiilerested in 
piuviiig that something they doirt like is 'Iwd" than the) are in g;itlieriiig 
d.iM toMipiHirt theii piacticcs. In the past, many studies which have used 
diildreii in scIkkjI settings have been highly ciitici/.ed. and often justi- 
hably. The iiiajoi eiititism ,selicK)l people have made of the educational 
reseaichei istliat the lesearch isn't pi.icticalaiid Intsiri helped theschmil. 
teacher, or student. Moie iniporiaiu. the te.icher fre(|uently suspects that 
"pure** rese.irch doesn't add to "science" as researchers claim, nor add to 
the Ixisic fund of knowledge. I hereniie. before the researcher contacts 
the 5clnH)ls. some attention imiM Ik- given to the cpiestion of why a scIkkjI 
would want him to i\u the reseaich in their setting. He should Ik- con- 
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viiiccd that ulijii he waiiLs to do is uorih the time leathers and children 
uill devoie to liis task. In addition, lie ^.liould luve dexeloped a rationale 
tliat lie can explain and defend to the .school pei sonnel. Once the research 
worker has developed a niiionalc. he should be auitious as to whai he 
promises the school. Frequently, in oin desire tu secure a school pupula- 
lion, we promise too much. I refer to this as ihe "Panacea Tmp" 
Kducatorshave tended to see each neu iimo\atiunas.i panacea for solving 
all their problems. Research workers sometimes have become over- 
enihusiasiicasto what their research .wlldenuHistiate if their hvpotheses 
are supported. 

My l>csi advice on how to initiate entrance into a school is to cdl the 
centnd office of the sch(H)l,to fiiul out if there are specific sets of proce- 
dures for gaining permission to conduci research in their schools. Once 
these procedures have been identified, follow iliein in e\er> respect. If it is 
not included, add lo whatever prcKedures the school may have personal 
tontart with each member in the hienirchy related to your research. For 
example, if the superintendeiu appro\es the project, be sure to coniar' 
the principal of the scluwl directly and in |>erson. Inexperienced re- 
searchers fre(|uentl> will by-pass the piincipal .uul contact ie;icliers di- 
rcctly. riiis is so serious a mistake thai 1 uould suspect feu researchers 
wiih any experience in coiiduciing research in schools will do so without 
fii'Nt securing the full cooiK-nition of the priiuipal. The principal sets the 
lone of ihe building and can do much to f.icilitate or. more imi)oi tantlv. 
subvert vour efforts. 

1 also am of the opinion thai the piincipal has the right lo know the 
full detailsof the rtwarch. and ii is the ubiigaiion of the researcher to fully 
inforunhe principal and answer any (picstions lie might have. Thestimu- 
lanon yon ina\ provide him In discussing your ideas with him may Ik* the 
ultimate service vou perforin f or tlie school. The priiuipafs role is often 
one of isolation, and he tends to enjoy discussing ideas with outside 
personnel. 

Following attainment of the piincipars appio\al. ask for and 
genib insist ii]K>n a meeting with the teacheis. pi eferabb after scIuh)! or 
at iheii lunch period, Fre(juciitlv.a principal will want totakeyou directh 
to a classroom at the time von are to do \uin research, and you will find 
that he has not asked the tcachei s permission to participate in the project, 
I would hesitate to conduct l eseaixh with chihiren from classes where the 
teacher has not l)een allowed to examine the instruments to Ix- adminis- 
tered and .isk c|uesiions as to the puipose of the research, .\faiu scIumiIs 
will lecpiire that \ou inform parents that their child is to take part in a 
re-search project A letter informing parents of the project containing a 
lu ief description of the go.ils and objcciivcts of -lie rcvsearch is usuallv all 
that is ncwssan . The phrase, ' ir vou have any (juestions regarding the 
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project, please call " is usually sufficient to secure the cooperation 

of most parents. 

Increasingly , teacher associations cuk\ unions are playinga decision 
matuig role in school districts. Be sure to check \\ ith the superintendent's 
office for guidelines as to the.union s role in the school district where you 
plan to conduct research. There ma> be limitations as to when you can 
meet with teachers and the teacher s role in participating with the re- 
search project, A check w ith the union representative will help clarify the 
local policy. 

A productive researcher on the west coast has very litUe problem 
obtaining large research populations — usuall) as often as he desires. The 
gcnerali/.ations I have stated above are drawn from his behavior. He 
always meets the principal in an unhurried manner and later meets wiUi 
the teachei or teachers to be invohed. If asked, he reports his general 
findings to the parent group or s|x;aks on an) other topic of the PTA's 
choice diat is w ithin his i ealm of competence. For one research project, he 
was able to secure without a single objection from principal, teacher, or 
parents, the entire fourth grade population, which was located in 22 
schools. 

Political pressure groups and pel theories 

If you are planning on trying out a departure from the school's 
regular piogiam— foi example a linguistic program in contrast to a 
phcmics progrcun-— take care to explore fully with the superintendent or 
his vcpiesentati\e whether or not you will cncountei emotional resis- 
tances due to pet theories in the disti ict, Occasionall) , a research worker 
will unknowingly find himself the center of a district-wide political cam- 
paign due to the fact that his research appears to counter a prejudice on 
the part of a minoi ity of the school |>opulation. For example, recenUy the 
director of a national study had to fl> to a school on the west coast and 
appear lx:fore an open schoolboard meeting because one of the stories in 
his material, which dealt with population data, was misinterpreted as 
family life education by one of the schoolboard members. Studies that 
(leal with leading conipi ehensioii may contain innocent paragraphs of 
content that are unacceptable to some local groups. In addition, there 
usually are other research projects lx;ing conducted in the school district. 
If these projects have an opposing orientation to the one you plan to 
conduc t, explore whethei your project will be percci\ed as a threat to the 
conductors of the original project. 

In one study in which I was involved, a schoolboard member was 
the coauthor with the leading super\isoi of an experimental reading 
seiies. lie was extieniely hostile towaid evaluation of his project and,* 
more important, peicei\ed allothei leseaich projects in reading as rivals. 
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School (listricipcisonricl licqucml) \mII i eject joui pnyei i asa proieetivc 
device to avoid an> and all piiblii-ielatioiis problems, riieiefoi e, iiis best 
to explore with them directi) uhethei oi not your project will cause 
conllict. The open discussion does tuo things: I) it often will uiake it 
possible for you to conduci \oui lesearch because it will give you the 
opportunit) to reassure them that \ou \\\\\ not causea problem; and 2) if 
your project posscssesa potential tineat tostmiegioup, )ou will be saved 
the cnd)arrnssmeiil of having \oui stud) used for political purpose by 
local newspapers or board members lunning in local elections. I have 
endured both experiences and enjo) ed neithei of t hem at that time nor in 
retrospect. 

The nature of the child sample 

Make an attempt to \ei il) the nature of the sot iai class status of the 
dnidren you are to test. Walk around the neighboihood and observe for 
vourself the general charactei istics of thesch;;'jl to determine whether the 
(uuward look of the homes meets^)oui expectation. You can often be 
fooled by the arithnietic mean the school has a\ailab!e. I hedata may look 
typical but ma> be a result ofdichotomous groups. Foi example, in an 
upper-iniddlcclass school, ue disimei ed a trailei ctuirt hidden behind a 
higli hedge The tnnler court children had just been tiansferred to the 
scliool. riie mean achie\einent le\el of two-thirds of the class was the 
eigluieth pen entile. The remaining one-thiid, the trailei court children, 
were achieving at the fortieth peicentile le\el. Combined, tlie total group 
mean score was in the 6()s. In anotliei stud), we were led to l)elieve we 
were dealing wiMi an m)nomicall) depressed population. W'lien we en- 
tered the classroom, man) chiklien did not look like the typical deprived 
grouj) Particularl), thcii clothes indicated a highei purchasing power. 
More important, die\ were exiremel) \erbal children and spokestandard 
vernacular. We {lisco\ered that a mimbcv of grailuate studems attending 
a medical schuol across tlie street enteied theii children in tlie.se classes. 
As you know, graduate students make \ery little money, .so tliechildren 
(pralified on economic grounds for these classes. 

Askirjg the j)rincipal whediei he uses any type of special grouping 
procedures n\i\) unco\cr some uni(|ue styles. In one scliool, we found six 
sections of the sixth grade. The piincipal indicated to us that he had 
assigned to a teacher who had nujored in psychology all of the more 
serious emotional problems. To a second clas.s, he had assigned a former 
high school teachei and had gi\en her the majoi beha\ioi -control prob- 
lems. The other four classes were, he reported, rathei placid, which 
gready reduced his work since this made it neccssai) foi him to deal with 
tm\y two teachers instead of six. 

In every case it is best to secure a laigei sample si/.e than needed for 
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yolii icscaich Iwunisi- the pioblcms listed cibo\c arc iisiiallv uncovered 
after the fact. In many cascb you wiW ha\e to discard data. 1 hus, having a 
larger sample than needed will save your study. 

Creating a positive climate 

If )ou arc to use only some classes in a school, offer to meet with the 
othei teachers to share )oui expertise. You need not discuss your experi- 
ment if know ledge of the ti eatment will interfere with >our stud) .Talking 
about new trends in reseaich as in reading oi language arts will help 
cieate a positi\e atmosphere foi >ou and your associates. Although the 
class oi children ma) not be read) foi )ou when )ou are read) toadminis- 
tei youi insli uments, do not mistake the teacher's scheduling problem as 
not caring about punciualil). Nothing is as disturbing to a teacher as 
ha\ing a class read) and then ha\ing the researcher arr[\e fne to ten 
minutes late. The same principle holds foi )oui appointment with 
teachers to desciibe your stud). Man) times )ou will ha\e to make the 
tiecisions about being on time b) insisting that )ou lea\e the principafs 
office even though he appears to be \ ei ) relaxed about the time. Recentl) , 
we weie to explain oiu .stud) to all the teachers from three schools at 1:00 
p.m. I he principals look iis out to lunch and we were all engaged in a 
li\el) discussion about schools. When we asked how long it would take to 
dri\e to the meeting place, the) said, "Not long." We grew increasingly 
anxious en route, as l;00 p.m. approached and the drive was much 
farUier ihcUi we expected. An ic) silence greeted us when we arrived late, 
and it continued to en\elop the room. We attempted to place a share of 
the blame on the principals b) noting that their generous Southern 
hospitalit) had caused us to be late, but that onl) made matters worse. It 
was more than an houi befoie we fell we had the group of teachers with 
us. 

Teachers desire all children to lake the instruments and share In 
the rewards. It is diflicult for leacheis to accept the notion of a random 
.sample. If it is feasible, administei )oui insli ument to the whole class and 
select )oin sample fiom the total group. In some studies where time 
consummg individual testing is done, this procedure is impractical. How- 
ex ei, if )ou institute a beha\ioi modification stud) in which rewards are 
administered to only part of the class, leachei s will desire )ou to pro\ide 
the whole da.ss with some t)pe of reward following the study. 

Space 

Most schools ha\e limited space for extra projects. If your research 
I eijiiii es a special i oom for administering your insti uments, y ou may find 
that you will conduct youi study in unique settings. I have administered 
iiistruinents in supply looms, the teachers* lounge, basements, picnic- 
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lunch bench aivas, hallways, cafclcriai, and in one cabc, ihc outer section 
of a bathroom, 'Vhc principal who is tiul) willing to cooperate with your 
study usually will find some place for you, e\en though you may have to 
operate on a highly flexible schedule. For example, our schedule for 
collecting language data in one large metiopolitan school was as follows: 
On Mondays and Wednesdays u e used the speech room aiKl,on Tuesdays 
and Thursdays, the auditorium. On Fi iday s \se had a corner hall which 
led to a dead end. This required our research assistant to move the 
equipment daily and on some days, u hen a **sui prise" program was held 
m the auditorium, to cancel her appointments. If you can use the total 
class, you have greatly simplified youi" prcxedures, but have not— as we 
shall see in the next section— totally soKed your problem. In some cases, 
we have reined a small trailei. By placing the trailer close to the school 
building, children can be brought to the testing site easily. Of course, the 
principal's permission should be obtained before renting a trailer. 

Administering the instruments 

I would suggest that in all possible instances the teacher should not 
administer your research instruments. Teachers are prone to want to 
assist children and frequently uill provide nonverbal cues to help them 
solve problems. On some occasions, teachers, wanting the child to meet 
with success, will provide direct help. And in some, hopefully rare, in- 
stances the teacher in his or her anxiety for the class to do well will 
actually provide children with answers. If this occurs, y our only option is 
to destroy the data. I maintain ue are guests in a teacher's classroom, and 
wliat we observe is privileged information. If a teacher intentionally helps 
children, we are in an ethical position of destroying the data but not the 
teacher. In all cases, I would recommend trained research technicians or 
the researcher himself administer all insti nments. In this respect, it is 
recommended that all pilot studies be conducted by the principal inves- 
tigator Usually, only you, the principal investigator, will be able to make 
decisions when emergencies arise. Once all the details have been worked 
out, others can collect the data for you. Teachers on leave make excellent 
research assistants. They ha\ e the ad\ antage of knowing the school and of 
being accepted by classroom teachers; they are familial with school regu- 
lations and are confident in handling individual oi groups of children. In 
some states, having a credentialed teacher as your research assistant 
reduces many school-related jMoblems due to legal constraints on who is 
legally able to supervise children. Teachers on leave readily respond to 
training and are usually very cooperative research associates. 

Occasionally, one finds himself in a classroom where the teacher 
feels highly threatened by the researcher. Fortunately, not too many 
teachers show abnormal behavior patterns, but in almost every school 
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(libtliu tlicic is at least one tcaclici uliosc room sliouki fx* a\oicle(l. Ik- 
sine to ask the piiiicipal if tliei c aie aii^ teachers in hisschoc)! who should 
not take part in the study for any reason. I do not suggest >ou proix? to 
fuui out the reason, accept the common excuse for pathology gi\eu by a 
principal: foi example, " Teacher X has had a great many family, health 
and/or person.il problems." 

In establishing rappoit with chiklien, ask them if they know what 
they are going to do with you. With young cliildien, we most frequently 
tell teacheisto inloim theii class they are going to play a game. One child, 
while walking Irom the first to the fourth llooi of the building, revealed 
that he was afraid we wei e going to electrocute him if he wasn t good. We 
pursued the issue, asking why he slunild think such a thing, and found 
that was what his teachei had infoimed the entire clasb. Due to the fact 
that we weie using head sets and tape recordeis, we had to eliminate the 
class from our sample. Clearly this teacher was disturbed with the 
childien— oi with us. Ineithei case, the data obtained would notbe\alid. 

The unexpected and interrupted 

We weie adnnnistering a self-concept measure to a second grade 
class in an economicall) depiessed aiea in Tennessee. Half-way through 
tlieadniinisti.uion, the piiucipalt.niieiiitotheioomand w hispered to the 
teacliei. riie teachei then came o\ei and asked if she could speak to me. 
M\ two research assistants asked the children to wait a minute. To make 
matters short, a local merchant of a Mexican restaurant had arrived 
unaiiiiouiiced at the school with fi ito chips, Mexican haLs, and a pinata 
filled with goodies. He insisted that he only had one-half hour. The 
principal said that the i est ol the school w as gathered around the flagpole, 
and he and the teachei wondered if I wanted the class I was testing to be 
left out of the activit) in older to finish the administration of the instru- 
ment. Thechoice was not a leseaich choice, it was a moral one. Did I have 
the right to continue the administi atioii of a measure and keep one class 
of childien lioni the excitement that was going on outside? We stopped 
the administi ation, went outside, ate friU) chips, donned Mexican hats, 
and watched the cliildi en sci amble foi the candy and toys once the pinata 
was broken. As we came back in, my graduate a.ssistant asked, "How valid 
will these tests be?" Cheered by the lelev.uice of my student^s (|uestion I 
said, "It's a leseaichable question, but aftei we allow the children to fmish 
the task, we'll probably throw these tests away. If you are interested, we 
can design a study to measure the impact of a peicei\ed-to-be-positive 
experience on childien's self-re|X)i ts." In othei words, most large-scale, 
so-called leseaich piojects in the school aie opportunities to identify and 
generate hypotheses, and should be perceived as such. 
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Majoi intei rupiions occui with such fi e(|ucnc> that I almost expect 
a fire drill to occur w hile ! am in a school. List Febniar> the principal did 
warn me he was to ha\e a drill that moi ning, so I w as prepared. But I w as 
not ready— nor w as the principal— for the four fire engines, one hook and 
ladder truck, anibulance, and ihiee |X)lice cars that arri\ed at the build- 
ing. His remark to the fireman bcais repeating. He said, "I'm sorry. I've 
only been here fi\e >ears and I keep forgetting to call the station to warn 
you of my drill. "The response by the fireman is of interest but is X rated. 

What I am sayingis that I, like the principal, have only been here a 
few years; and research in the schools is a source of continuous surprise. 
Classes will be on field trips, at a puppet pla>, in another room, or listening 
to a story in ihe libiar>. Much more difficult to handle is the situation in 
which theciass ma> ha\ejustgone thiough a negati\e experience with the 
teacher. A colleague of mine was administering an experimental foi m of a 
reading test in a classroom in w hich a piano had been pushed against the 
two sides of one corner to make a triangulai space. While he was handing 
oiu t he iiistnnnent, a small \oice came from that space: "Sir, I don't have a 
copy," Rccenil) we went to a rural first grade classroom to conduct a trial 
session of oui language curriculum. V\'e noticed that a young girl was 
silting in a pool of liquid, and the leachei had not excused her nor offered 
her any assistance. When \ve asked the teacher what had happened, she 
said, "She always wets during music," I was rather surprised wlien my 
colleague (juieil) said,"Pei haps that was the most appropriate response to 
the lesson," 

Leaving the school 

After youi research project has been completed, a letter should be 
sen! to all parents w hose children took part in the stud>. The letter should 
be written in la> terms summaii/ing the major findings and thanking 
parents foi their cooperation. This action will do much to insure your 
continuing acceptance b) the schools and parent coinmunit) for future 
research projects. Occasionall) a paieni ma) wish lo know moieaboutthe 
research, and )oui phone mimbei included in ihelettei with an in\itation 
to (all for additional information will .satisfj this need. The letters can be 
sent home via the schools oi appeal in the schoofs newsletter, if one exists. 
I n one stud) , w e w ere able to maintain contact w ith o\ ei 90 percent of our 
research population (ner a period of four >ears. In some cases, parents 
who had moved from the school district informed us of theii new loca- 
uons and volunteered their child's continuance in the study. 

I also recommend that an> joui nal article should footnote thanks 
to the school district in\ol\ed. In some rare cases, where the data may 
reflect unfavorabi) upon the schools, peusonnel should be given the 
option as to whether they are identified specifically. 
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Reporting: the last major step in the research act 



BRUCE TONK 
Editonal A suwiate 
Reading Research Qiiartetly 



The importance oj effective reporting 

The most coiiipclliiig, original, and caicfiilly executed research 
will ha\e minimal impact until its findings are communicated. AUIiough 
research is a uortli> aciu\it> in and of itself, its primary aim is to increase 
man's knowledge; and it cainiot do Uiat unless it is reported clearly. It 
must be logically oigani/ed, complete, ind clearl> and carefully written. 
So essential is the need to i eport research effecti\el> that it is argued here 
that reporting is the last major step of die research act and that die impact 
o( all die \ ital steps in conducting good i esearc li ultimately depends on it. 

Although reseaich is often reported verbally, more often it is 
written; and diis chaptei focuses on that method. Most of die points 
considered, how e\ei.aie applicable tobodi forms of reporting. A discus- 
sion of how audience awareness (an help insure a good report precedes a 
relati\ely piesciipti\e analysis of die essential sections a complete report 
should include. The chaptei concludes with the consideration of some 
miscellaneous concerns, most of which are related to specific style prob- 
lems. Much of the iiiateiial in the chapter has accrued from encounters 
with inaiiiisci ipts on reading research that had been criti(iued and edited 
by expeits in die field. ConscHjuendy, this discussion fiequendy clarifies 
what a repoi t ought to do by pointing out what some researchers unfor- 
innately do instead. If that treatmeiiL is reminiscent of composition 
com ses, considei that icseaicli lepoiting is essentially just good exposi- 
tion, the goal of all freshman composition teachers and manuals. 
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Audience axoareness 



Adopting the reader's point of view 

Ecliiorial analysis of research nianuscripis reveals how closely re- 
lated dear thinking is to dear lepoi ting That is not to imply that when the 
researcher reports ineffectivd), he does not understand his material. 
Rather, intense involvement in the material sometimes betrays the re- 
searcher when it is time to write the report. The researcher is so dose to 
the material that it is e.\tremel> difficult for him to remember the audi- 
ence s uninformed condition and to recogni/.e the need to arrange the 
report logically from that point of view. Also, the researcher, who is so 
familiar with the subjects, materials, and procedures, may gloss over or 
omit details the reader needs in order to understand the report. Some- 
tnnes the reader's understanding is muddled because synthesis that oc- 
curred in the researchers mind during his activity goes unexplained in 

thcM'ciX)rt. These problems are avoidedb) adopting an awareness of one's 
audience. 

Selecting an audience and a vehicle 

The audience to be considered must be selected first, however; and 
that sdection is the result of the thinking the researcher does in choosing a 
vehide. This sdection will be effected by the material itsdf and by the 
audience the writer wants to reach uith that material. A publisher for the 
practitioner audience, for example, would be much more interested in a 
study with immediate classroom implications than in one with implica- 
tions primarily for future research. And the practitioner audience would 
be more interested in those implications than in the design or implemen- 
tation of the research. Thus a report u ritten for such a journal may need 
to frame that rdevance from the outset. An audience of fdlow research- 
ers, however, is usuall> suspicious of a report that begins by stressing 
implications; and a journal published foi such an audience ma) require a 
presentation that delays dealing uiih the implications until the reader has 
a chance to examine the study and its findings. 

liven within audiences, specific publications may treat reports dif- 
ferently, hi so doinjj, they tend to cultivate a more spedfic audience, 
ujndi comes to expect the kind of reporting that attracted it in the first 
place. Sometimes a publication s "st)le" of reporting is controlled by its 
need to man ) its audience's prioiities to the economic factors in publish- 
ing. Such considerations may affect the choice of subject matter as well as 
how extensively it can be treated. 

11ie researcher's concern in all of this is one of awareness, After 
determining the appropriate audience, the l eseaichei must target a logi- 
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cal publication as a vehicle f oi the report. The process then reverses. The 
nature of the vehicle selected and its particular audience act as lexers on 
the reporter and may modify the intentions for reporting the research. 

By carefully selecting a vehicle and by then preparing the report 
specifically for submission to that publication, a researcher considerably 
improves the chances of getting a leport of good research published. One 
way to do this is to examine carefully both the kinds of reports a publica- 
tion has been printing and any stated intentions of its editors to publish 
broader content. Many journals have manuals or guides that give authors 
deal indication of the kind of material the editors accept. Such guides also 
detail procedures for submission that, if heeded, can insure the considera- 
tion of the manuscript. 

A publication's audience can be determined from information 
supplied by the publisher, but some simple reasoning can also prepare the 
researcher to write with an appropriate audience in mind. A research 
oriented publication, such as T/wJoimial of Reading Behavior or {h(: Reading 
Research Qiiarlerly, will have an audience of researchers, college teachers, 
specialists, and giaduate students from both the specific field and from 
numerous related fields. Though varied, such an audience has a common 
sophisticated interest in the field, and it is not difficult to determine at 
what level explanations must Ixjgin and what the audience in general 
expects from the report. At the same time, the diversity of such an 
audience underlines the need for clarity and precision so that each reader 
can easily identify the relevance the i esearch reported has for her or him. 

Outlining the essentials with the audience in mind 

Audience awareness stresses that the organisation of a research 
re|)ort ought to be structured from the reader's point of view. As in any 
good essay, the readei expects a general structure that moves from an 
introduction thiough a development to a conclusion. Within this struc- 
ture, the readers of research journals expect to learn what was done, why 
it was done, how it was done, and what the outcome was — and preferably 
in that order. Thus, the carefiil delineation of a problem, the develop- 
ment of a rationale, the execution of an appropriate design, and the 
analysis of results can help guarantee a well organized report; for the 
author has full control of his material from the outset. A logical considera- 
tion of how the reader can best comprehend this material leads to mean- 
ingful organi/.ation, which in turn facilitates clear writing. A strong orga- 
nization especially guarantees effective transition between parts and also 
between paragraphs. 

In most instances where good intentions to cover all of the essen- 
tials end up in muddled reporting, the researcher would have been 
greatly aided by outlining the report before beginning writing. Bearing in 
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niiiicl the logical ic{|iiiicmciu that the exposition begin, cle\elop, and 
(onclucle, the repoiiei of research can iranblaie the join nalislic require- 
ments of what, why, and how into the more specific sections that a 
research report should include: a brief but direct explanation of exacUy 
what the stud) did; a statement of the problem and a discussion of the 
background and rationale for the study ; a re\iew of related literature; any 
hypotheses being studied, a description of the procedures used including 
a full account of the instruments used, the statistical technicjues employed, 
the population studied, and any steps used in collecting the data; a 
discussion of the limitations and assumptions of the stud) ; a presentation 
of the findings; interpretations of the findings; conclusions drawn on the 
findings; and a discussion.of the implications of the findings, 

There are other acceptable frameworks; but to be complete, a 
reporter needs to cover all of those elements that apply to his study in 
some logical way. Various t> pes of research will modify the requirements 
of a good report discussed in the following section, which is most closely 
aligned with repoi ting experimental studies. Not e\ ery report would have 
a separate section for each of the elements just listed; nor would every 
manuscript prescni them in the order gi\en. The amount of fiexibility in 
the prescription is controlled b) the t)pe of study, by its content, and by 
the needs of the selected audience, The important thing is that the 
researcher report clearl) and completel). using framework that is 
logical from the reader's point of view, 

Elements iu a good report 

Good organization pro\ides the framework for logical exposition 
and helps set up natural transition, but it cannot guarantee clear descrip- 
tion within those parts. Each basic section that an effecti\e research report 
normally includes should offer specific information, consequently, an 
analytic look at reporting, part b) pait, may be useful to the reporter. 

Introducing the study 

Most researchers understand the need foi an introduction to their 
reports; but surprisingi) often it is not theii stud) they introduce to the 
reader but the general topic area in which the study takes place. This kind 
of background can be significant and a legitimate part of an introduction, 
but only after the readei is offered a statement of exacti) what the study 
being reported did and is told about the problem that led to a rationale for 
the research. Some authors list h> potheses in the introduction, but usually 
they are given in a later section detailing the stud). These primary ele- 
ments of an introduction tell the reader what to expect from die report, 
whether he needs and w ants to i cad it, and how the background material, 
which should follow\ relates to the topic at hand. 
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Identification oj the .study at hand. It seems obvious thai the reader 
needs to knou right of f uhat the researcher did, but it is not unusual to 
come across manuscripts — and even pubhshed articles — in which the 
reader is strung along for 500 to 3,000 words before he is given the 
information. When the reader finall) gels to this \ilal information, it is 
necessary for him to reread all the preceding pages in order to restructure 
the material with that focus. In a sense, the reader is doing this for the 
author, who failed to consider the reader's point of view. 

Some authors argue that an accompanying abstract of their study 
suffices fur an introductory explanation of the study , but a good abstract is 
written from a completed report, which is adequately introduced. The 
abstract is written in a \ery turgid, special language style and serves a 
different reader need than does the cuinpleie report; it is no more an 
introduction than is the title. 

The rationale. In the introductory section, the reporter must con- 
\e> the importance of his study as it relates to any problems, questions, 
and theories which moli\aied it. In this section of a report— sometimes 
called the ^'background" ol* the research— the researcher needs to con- 
vince the 1 eadei that his study was worth doing. Otherwise, the reader can 
easily conclude that the merit for having conducted the study is either 
suspect or at least questionable. Such a discussion will occur as a part of 
one*s rationale. 

Some authors prefer to use the discussion of the rationale at the 
very beginning of the report followed by the initial description of the 
study. Others handle the rationale in a discreet subsection. Since the 
rationale is the resulting heart of the background material, it frequently 
involves some of the majoi related sources. If such sources cannot be 
treated separately from othei related literature, the reporter is forced to 
combine the background with the related literature. In such a case, the 
reporter must wi itc exceptionally clearly , identifying the rationale within 
the other b.ickground material. 

Definition of terms. When the terms in a study are unusual to the 
audience, applied to the study in any special way, affected by unresolved 
issues in the held, or unusually complicated, an early section defining 
terms will avoid confusion foi the reader throughout the report. 

A key for the reader. In introducing the study, the author has the 
option of using the seemingly ob\ious but surefire technique of giving his 
leadci a short itinerary of the ground that he will cover. A paragraph can 
key the reader to the majoi points to be considered and track him through 
the repon. To do this, the author must have necessarily outlined the 
report, knowing clearly how it will de\elop and where the emphases will 
fall. Many authors avoid this approach as unworthy of polished exposi- 
tion, but a report can be written without sounding as though it has been 
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taken from an original proposal foi the bUidy, If, houevcr, the skeletal 
words "This article reports a stud) uliicli , . are followed by a billing so 
accurate and balanced that the readei ne\ ei feels off the track throughout 
the article, the author could have done much worse. 

Reviewing the literature 

Of all the elements in a report, the section on related research or 
literature often pro\es the greatest challenge to the researcher^s writing 
ability. Many reports prefei to include related literature as a part of a 
hmger introduction; but particulai 1) u hen there is a great deal of material 
to 1)0 covered, theanalysisof it ma> stand as a subsection of the manuscript 
by itself. Some authors handle the ie\ie\\ of i elated research as a begin- 
ning of the body of their report, and adept analysts ma) skillfully use a 
miinber of studies later in the rejxjrt to help delineate aspects of their own 
research. 

In presenting related research, the writer must call on his powers 
of selection and analysis in order to place the study being reported clearly 
against thebac kground and in order to pio\ide meaningful oigani/ation 
for the material in this ,section. This perspective is blurred if the author 
feels that, having selected a study foi examination in the com se of his own 
reading, he is obligated to summaii/e it for his readers whether it is 
relevant or not. Some authors find it difficult to select efl'ectively for this 
section because their interest— or e\en their research— is broader than 
the areas they have chosen for their report. Such authors seem to be 
spurred to include as niuch tangential information as possible. This 
problem occurs frequently uhen unpublished authors adapt their disser- 
tations for publication, A paddyd section of i elated literature would seem 
to confuse the reader about the relation of the study at hand to the 
background material. 

A weak related literature section of a report can result when an 
author exercises no analytic levei on the material. Consequently, the 
related studies are presented in chronological oidei oi are merely jux- 
ta|)ose{| in no apparent ordei , One feels that he is reading a collection of 
loosely related absti acts. The l)est i ev iews of litei ature i esult from cai el ul 
selection and are presented with an organization, that is a product of the 
authors synthesis of what the related studies mean in terjiis of the study 
being reported \lmost always, the author w ho handles This section of the 
re|X)rt skillfully also displays skillful w riting and cleai thinking through- 
out the report. 

Another significant problem noted in the reporting of related 
literature is the failure to explain studies fully and/or clearly enough for 
the reader to grasp them. No study i elevaiit enough to include should be 




^ ^'^ lieporUtig research 



120 



presented to the leadei in an inicleai manner; hut in too many reports, 
even major rehited studies are inadequately explained. 



Describing the study in detail 

The description of the study being reported often conies under 
one major heading. Such a section may include subsections on design, 
subjects, materials, and procedures, and a finther breakdown may be 
used in describing these. What is important is that the description of the 
study be complete and logically presented, so that when the reader has 
finished reading it, he understands what was done. While the nature of 
the study being report d may account foi \ariations in its description, it 
cannot excuse a lack of completeness oi ambiguity. This section, above all, 
nml he crystal clear to the reader and must not raise any major (juestions. The 
description of the stiuly must gi\e enough basic information for die 
reader to determine what is involved in replicating it. 

Sometinies replication in\ol\cs a (|uantity of detail that is available 
elsewhere: the reader can be referred to the exact content of a long 
(juestionnaire, foi example. 'J here is Ju)we\ei , nothing more frustrating 
for a reader than to Ix; persuaded that a study is \alid and then to find 
important information missing in the description of its design, materials, 
instiumenis, subjects, or procedures. The writer should never allow the 
reader to doubt the integrity of honest research Ixrcaiise of vague or 
missing descriptions. Infoimation muddled oi omitted may include de- 
tails that would icvealaninadeciuatesainpleoran untreated variable. The 
careful reader is not fooled when an author is, as one reviewer of a 
research manuscript once expressed it, "deplorably Vavalier in providing 
less than the minimum information about the subjects used." 

Some researchers use designs, instruments, and statistical ap- 
pioaches which are \ei y sophisticated. Describing those aspects of such 
studies foi the average i eadei of a research journal can be very difficiilt. 
Unless the researchei is communicating through a \ehicle basically lim- 
ited to readers at his own level of sophistication, however, the reporter is 
obligated to explain those aspects of his research so that a majority of his 
leaders will understand his report. Tables and figures can* be used to 
simplify the explanation, but the leporter must give the reader enough 
information in his exposition to read the tables and figures. Also, the 
reader can Ix} referred to sources which detail the type of design. 

' A checklist ol the elements in a complete section detailing a study 
includes careful descriptions of: 
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the hypotheses, if not used earlier in the report 
the resetirch design 

the procedures followed in implementing the design 

all instruments iised in the study 

the statistical techniques employed 

the population studied and samples used 

all steps followed in collecting the data 

Considering the study's limitations 

One of the surest signs of a careful researcher is the reporter who 
handles the limitations of his study openly. This section should be 
thorough and, after consideiiug ail the aspects of the design and proce- 
dures, include any weaknesses in sa.iiples, instruments, oi design. Uncon- 
trolled variables and any assiuuptions made in the study should be noted. 

The reliable reportei will resist the temptation to dull the edge of 
this self-examination with \agueuess; yet many authors are unable to 
broach this section at all, and when prompted to do so will interpret 
limitations as ineicl) considering the limitations of extending the findings 
as implications. Wise icscaichers stiengthcn the real worth of their work 
with candid objecti\it). On the basis of an open section on limitations, it 
can more readily be determined whethei, with the acknowledged (|ual- 
ifications, the repoi t of the i esearch mei its attention. Not including such a 
section docs not mean that the limitations will not be considered without 
the authorN acknowledgement, and the editoi, ie\iewer, oi reader who 
nuist make thisc\aluation alone is justifiably <|uite waiy of the leseaicher 
w ho at know ledges no flaw s in his oi hei w oi k, suggesting that it is perfect. 

There is no absolute prescription for placing limitations in a re- 
port. They can become a subsection oi a separate unit in the report. 
Sometimes they lollow the detailed description of the study. Often they 
are discussed aftei the findings aie presented and before sub.se<|uent 
analysis ol'the findings. Some authois use them before ox aftei a discus- 
sion of the implications, and although thej aie \aliuly related to this 
elemeiu of the icpoit, they tend to make a weak c(mclusion to a report 
when presented last. 

Kvaluating the results 

()ne ol the majoi problems that editors of research reports en- 
counter is in helping aulhois clcai up the e\aluati\e discussions in their 
reports. It is partitularly crucial that an authoi distinguish among 
findings and intei prctations, conclusions, and implications in the report. 
\Ianv writers (onfuse these and thiow them into a fwt fmmri under the 
subtitle "Discussion.** Too often the result is an uninteipretable mish 
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mash. An author woukl be well a(l\ise(l to separate these elements and to 
present e.ich as a distinct segment of the research report. 

Findings and interpreiatwns. The findings are often presented at 
the end of die major section describing the study, and the fact that they 
are not an e\aluative element recommends this placement. When the 
findings are long and include numerous tables, they can be presented as a 
major .section by themselves. But because all of the evaluative elements of 
the report are based on them, many authors begin their evaluative section 
with them. It is essential in doing so that the researcher let his findings 
stand free of his evaluative analysis, however. 

In the section on findings, the writer simply reports what was 
found. Findings grow directly out of the implementation of the study. 
After presenting the findings, the author will the.^ probably want to offer 
some explanation of w hy the study found w hat it did. These are interpre- 
tations of the findings and may logically appear in the same subsection. 

Conclusions. Conclusions are developed from fiiiidings. They are 
the generalizations on the subject of the study that one can draw based on 
what has Ixjen discovered. 

Im/diaitwns. Implications are an authors estimate of what his 
findings and conclusions mean. The wiiter is, in effect, saying that if his 
findings and conclusions are accinate and acceptable, they then have 
certain implications foi practice, for future lesearch, oi for theory devel- 
opment. Fieqnently authors equate implications with recommendations 
foi futiu e research, w hich is but a part of this opportunity to evaluate the 
findings. The researchei should determine if there are other implicatioirs 
as well. 

As one experienced editor of reading research reports points out, 
The exposition of each of these elements of the report should force the 
author to come to giips with himself and his study and to face sever.il 
successive moments of truth." 

Some Specifics on xuriling 

Once die w ritei has selected material so that it is clearly focu,sed on 
die topic and cUidience, has oidei ed his material in a logically infoi mativc 
way, and has de\eloped it completely* he will want to be sure his style is 
precise and easy to read and imdcrstand. Specific style problems fre- 
i|uently relate to naiiowei aspects of the language, such as word choice, 
sentence stiucune, and the way clauses, sentences, and ideas are con- 
nected*. 

Writing precisely and clearly 

Diction. Although the nature of the content in research jom nals 
frequently is highly statistical, generally the diction of the bettei written 
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articles is clcai and iiinplc without talking down to a knowledgeable 
audience. The terminology is usually standard to people in the field and 
need not, therefore, always be defined. If, however, the terms are new or 
perhaps are being used in a unique or different manner, definitions are 
called for. Perhaps the best advice might be, "When in doubt, define.'* 
Once this question of defining terms is settled, the writing in any publica- 
tion should be clear to a neophyte. 

Ideas that the language cannot handle with single terms have to be 
expressed with phrases; but occasionally a w riter will abuse this language 
phenomenon and indulge in what a reviewer recently referred to as 
**glamorous terminology of the sort that theoretical sociologists enjoy 
using." An analysis of such diction reveals that the phrases it creates are 
often metaphors for ideas easily presented with simpler terms. Such 
diction is a kind of academic "pooi poetry" which few readers are eager to 
explicate. When a report writer cannot resist creating unnecessary jargon, 
he has the obligation to clarify. But writers sometimes fail to define even 
necessary terms for the reader when they are first used and may fail to use 
the same terms throughout the rejxjrt. Sometimes the same term is used 
with several di.stinct meanings. If the signals are mixed or unclear, com- 
nnmication will not take place. 

Syntax. Frequently, problems with sentence structure in research 
manuscripts relate to length and to the way ideas are connected. Long 
sentences die often weak and suggest that a real problem is a frugal 
attitude on the part of the writer toward all the information he has 
gathered. In a kind of fren/y to get every thing possible into a published 
article, a reporter may reveal a startling lack of understanding of the 
relevance of syntax to ideas. The number of misplaced and dangling or 
floating modifiers that areencounteied in some manuscripts suggests that 
many writers expect their readers to do all the thinking by relating ideas 
that are merely juxtaposed. 

Careless selection of joiners or omission of them altogether relates 
ideas obli(|uely to their real association, and the writer muffs the oppor- 
tunity to communicate with exactness. Many writeis give far too little 
thought to the precision of »heii conjunctions and prepositions, The 
joiner with the precise shade of meaning needed is overlooked in the 
writer's penchant for another word, which, overworked, creates 
monot(my as well as a lack of clarity . Often in research reporting, numeri- 
cal se(|uence is used rather than a moie analytic transition between ele- 
ments in a series of ideas. Writers might challenge their inclination to 
enumerate to be sure that counting is the best way to show how the ideas 
being counted relate. 

Actually, any lack of appropriate grammatical relationship loses 
editors before le.ideis, and uncle*u ideas should be either adequately 
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related to oi omitted from the exposition before publication. It is not 
surprising that-— on being asked to rewrite uncleai passages— many au- 
thors are quickly convinced that the sentences are not worth saving. 
Occasionally, an unclear clause in the original manuscript ends up as an 
important paragraph of its own in the rewrite. 

Certainly, communicating ideas cleai ly is the objective of research 
reporting — not the display of sophisticated essay form. In this respect, 
writing a research report is different from literary writing. Although 
excellent exposition has its place in a research report, the purpose of 
research reporting must be to help the reader to understand the research 
and to evaluate its quality from the rejx^rt. In a clearly written article — 
whether it is polished in style oi not — reader attention is immediately 
focused on the specifics of the content. 

Accuracy, Although accuracy is not really a matter of style, it is the 
most universal specific requirement enforced on a manuscript. One dis- 
cernible mistake in a published report can undermine the reader's entire 
confidence in the study because, as a rev iewer has put it, it "may signal the 
presence of other careless errors that arc not discoverable in the report." 
Accuiacy requires vigilance. Every time material is copied — from notes or 
computer printout to handwritten drafts to typewritten drafts to final 
manuscripts — one must guard against the potential for inaccuracy. 

The highest potential for outright inaccurate information in any 
manuscript is often in iti> reference list. I f this part of a report is to serve its 
purpose as a tool for die reader, it must be accurate. It is the researcher's 
responsibility not to misdirect his reader by misspelling an author's name 
or by giving the wrong date, volume number, or page numbers. Such 
errors fre(iuendy occur in submitted manuscripts, and they shake one's 
confidence in the full report they document. 

Using subheads 

Many writers are perplexed by the need to identify the parts of 
their reports uith internal headlines called subheads. These are not only 
attractive typographic devices, but they act as guideposts to the reader. 
Individual publications have unique formats which use different 
typefaces for various levels of subheads. A good reporter will know the 
various levels of subheads used by the publication he has selected as a 
target and will exploit the subhead device to reveal organization and to 
help the readei grasp the material. One way of doing this is to be sure the 
subheads used at each level are parallel in both importance and syntax. 
Subheads are sometimes merely labels of the standard parts or elements 
of the report, but die more inforinati\e subhead will also bill some of the 
specifics of the content in the segment it identifies. Since subheads are 
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(Ii\i(leib, not joiiicib, the uiitci slunild not id) on buhlicacU to rephice 
transition between parts. 

The use (jf subhetuU offers the writei an excellent oppoilunil) to 
e\ahiate hib organi/.atioii. I f, after completing the first draft of a report, 
the repoitei lisi.s the subheads used, indenting unifoi ml) with each le\el, 
he uill end up w iih a topical oudine of the i epoi t. A quick examination of 
this outline will ie\eal 1) whethei the lepoit is logicall) ordered, 
2)wheihei the paitsofthe repoit undei majoi subhea(ls are adequatel) 
parallel and those pans piesented as a bieakdown of each major part aie 
adequatel) paiallel, and 3) whethei the parts of the lepoit uhicji should 
be parallel add iq) to convincing units. 

The integrity of reporting 

Writing stvie is highly indi\idualistic, and this fact guarantees that 
careful coiisidei ation of the basic elements of reseaich wiiting will result 
in bettei lepoits wiitten in unique styles. Vet, the lese.uch report wiiicr 
has the obligation to be cleai and infoi mati\e, to be precise and accurate, 
and to be thoiough but rele\ani and concise. To be less than that is to fail 
himself, his iese.nch,and his reader. A good lepoi t will notco\er up for a 
poor study, but a weak report can bui'v the best research. 
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